The digital age has declared data the new oil, and in this landscape, Harvard University is emerging as a significant player. The institution recently launched a massive dataset comprising nearly one million public domain books, specifically designed for training AI models. Funded by tech giants Microsoft and OpenAI, this initiative falls under the umbrella of the newly established Institutional Data Initiative. The dataset draws from Google Books’ scanned collection, focusing on works old enough to have entered the public domain, free from copyright restrictions.
A Diverse Literary Collection Fuels AI Development
This expansive dataset, as reported by Wired, encompasses a rich tapestry of literary works. From timeless classics by Shakespeare, Dickens, and Dante to niche Czech mathematics textbooks and concise Welsh dictionaries, the collection offers a diverse range of textual material. This breadth of content is crucial, as copyright generally protects works for the author’s lifetime plus 70 years.
The Data Dilemma: Balancing AI Progress with Copyright Concerns
Foundational language models, such as ChatGPT, rely heavily on vast quantities of high-quality text data for training. The more data these models absorb, the more effectively they can mimic human language and provide information. However, this insatiable need for data has created legal and ethical challenges, particularly regarding copyright infringement.
Several publishers, including the Wall Street Journal and the New York Times, have filed lawsuits against OpenAI and its competitor, Perplexity, alleging unauthorized use of their copyrighted content. Defenders of AI companies often argue that humans themselves learn and create by synthesizing information from various sources, and AI operates similarly. This argument, however, overlooks the sheer scale and speed at which AI processes data, far exceeding human capabilities. The Wall Street Journal’s lawsuit against Perplexity specifically accuses the startup of “massive scale” copying.
Another argument posits that any content openly available on the web is fair game, and that chatbot users, not the AI companies, are accessing copyrighted material through their prompts. This analogy compares chatbots to web browsers, but the legal implications remain contested.
In response to these criticisms, OpenAI has negotiated agreements with some content providers, while Perplexity has introduced an ad-supported partner program with publishers. These actions, however, appear reluctant and highlight the ongoing tension.
The Shrinking Pool of Accessible Data
Simultaneously, commonly used web sources, often already incorporated into training datasets, are increasingly restricting access to their data. Platforms like Reddit and X (formerly Twitter) recognize the immense value of their real-time data, particularly for enhancing foundational models with current information. Reddit generates substantial revenue by licensing its vast collection of subreddits and comments to Google for AI training. Elon Musk’s X has an exclusive agreement with his xAI company, granting access to the platform’s content for both training and information retrieval. This protective stance contrasts sharply with the perceived lack of value assigned to content from media publishers, which is often treated as freely available.
Harvard’s Contribution: A Step Towards Ethical AI Development
While one million books won’t fully satisfy the training demands of any AI company, especially considering the historical nature of the texts, Harvard’s dataset offers a valuable resource. It provides a legally sound starting point for training foundational models, allowing AI companies to avoid potential copyright issues. This contribution is particularly important as AI companies strive to differentiate themselves through access to unique, high-quality data, ensuring the development of diverse and innovative AI models.
