AI needs a 'Stargate for Data,' a new viral essay argues. Here's what made my eyes bug out
Former OpenAI researcher Will DePue argues AI's next bottleneck is data—not compute. His proposed fixes include buying companies purely for their data, and getting firms to stop deleting theirs.
Today’s frontier AI models—ChatGPT, Claude, Gemini—owe much of their success to the public internet. For years, AI companies feasted on billions of web pages collected through archives like Common Crawl and massive datasets such as The Pile.
Then, of course, came reinforcement learning from human feedback, and other post-training techniques, that rely on human-generated data designed to teach AI models how to reason, use tools, and complete real-world work. That demand fueled a fast-growing ecosystem of companies that recruit experts, generate training data, and evaluate frontier models, from Scale AI to startups like Mercor, Turing, and Handshake.
But a new viral essay published yesterday says none of that will be enough, and what we need is massive new effort — equivalent to the current AI data center build out — to acquire the data needed to keep scaling frontier AI, with proposals that range from turning off corporate deletion policies to buying companies for their data.
The next bottleneck
Former OpenAI researcher Will DePue, who left the company in April, likens the need for more data to the massive AI infrastructure buildout currently symbolized by OpenAI’s Stargate project, in which Big Tech and top AI companies are predicted to spend trillions over the next decade to erect supersized data centers that power tens to hundreds of thousands of GPUs to train and run frontier AI models.
DePue argues that AI’s next major bottleneck won’t be computing power—it will be data. He predicts that by the end of the decade, leading AI labs could be spending more than $100 billion a year acquiring and creating the data needed to train future models. That, he explained, requires a massive “Stargate for Data”-type initiative.
As researchers increased both the size of AI models and the amount of training data, he explains, capabilities improved steadily, even if each additional gain required exponentially more resources.
Now, DePue argues, the era when the public internet provided an effectively limitless supply of training data is ending. Compute is no longer the primary constraint. If we want economic progress and scientific acceleration, access to high-quality data is the key to breaking through bottlenecks.
Therefore, he writes, “We need to see data collection as imperative, deserving the same civilizational ambition we’ve given compute.”
It’s remarkable how DePue matter-of-factly admits how much AI progress owes to “the blessing” of the internet, which he calls “a one-time civilizational subsidy to deep learning, decades of unintentional accumulation of a perfect dataset: every book, blog post, image, video, paper, discussion, etc. all digitized and freely available.”
I must say that’s probably not how the plaintiffs in the dozens of copyright lawsuits against AI companies would characterize it, but hey — let’s just leave that there for now. DePue’s point is that we’re running out of it, saying that there are only about 300 trillion tokens of useful public human text left, and the internet doesn’t produce nearly enough new high-quality data to match what scaling demands. The rise of reinforcement learning gave AI labs a temporary reprieve, he added, because models could improve by solving math and coding problems that were abundant online and easy to grade automatically. But now he argues even those datasets are being exhausted.
Going forward, the best data — rather than chips or model architecture — will become a frontier AI company’s most important competitive advantage. OpenAI’s strength in mathematics? Anthropic’s performance in cybersecurity? It’s no accident. Labs are increasingly relying on proprietary datasets, expert-generated training data and custom reinforcement learning tasks, which are differentiating their models.
Okay. Now let’s get to where my eyes start bugging out. DePue’s proposals for closing the data gap get immediately into consent, market power, and his belief that maximizing economic automation, as fast as possible, is just good. He says he’ll break them down in a later post, but it’s easy to see where he’s going.
