GPT-2 shows scale, not labeled data, now sets the pace in language AI

OpenAI trained a 1.5 billion parameter model on raw web text and kept it back. Language ability now comes from scale, and labs decide who gets it.
A language model trained on nothing but raw web text has set records on seven of the eight language modeling benchmarks it was tested on, none of which it was trained for. OpenAI unveiled the 1.5 billion parameter system, GPT-2, on 14 February and kept it back, publishing only a much smaller version and a technical paper over concerns that the full model could mass-produce misleading text.
The withheld weights draw attention; the recipe moves money. GPT-2 is a direct scale-up of the model OpenAI described last June, with more than ten times the parameters and more than ten times the data: 40 gigabytes of text from 8 million web pages. It saw no labeled examples and no task-specific training. If general language ability keeps rising with compute and unlabeled text, the scarce input in language AI stops being the annotated data set a start-up spent two years building. It becomes the budget to train at scale, and the labs holding that budget also decide who gets the result.
The scores show how fast the baseline moves. On the Winograd Schema Challenge, a test of common-sense reasoning, GPT-2 reached 70.7% accuracy against a previous record of 63.7% without seeing a single training example from the test. In November Google open-sourced BERT, a pre-trained model that set records on eleven language tasks and can be tuned into a question answering system in about 30 minutes on a single Cloud TPU. Both results rest on the same economics: pre-train once at great cost, then adapt cheaply. They part ways on access. Google published its weights, while OpenAI kept its largest model, its data set and its training code in-house and said it would revisit the choice in six months.
The arithmetic of a language product changes with it. A team whose edge is a model trained on its own labeled data now competes with a baseline that improves every few months at someone else's expense. Durable positions sit elsewhere: proprietary data from a workflow the general model never sees, distribution into customers who pay for an outcome, and the engineering that turns a pre-trained model into a dependable product. Fanaura looks for AI-first teams that treat the best available model as a part they will replace, and plan costs on the assumption that it keeps getting cheaper, the bet at the center of our venture thesis.
The objection deserves weight. GPT-2 drifts off topic after a few paragraphs, and on reading comprehension, summarization and translation its scores remain far below systems built for those tasks. Replication will settle it: if other groups reproduce comparable results from the published paper within months, scale has become a method anyone with capital can run, and the advantage passes to whoever buys compute cheapest and ships on top. Washington also has a say in who shares such models, through the export-control review that already names machine learning.
Language AI now behaves like an industry with a cost of capital, where the frontier belongs to whoever pays for scale and the margin to whoever builds on it best.
Building an AI-native company? Apply in five minutes. →
More from Fanaura
Building a language product on proprietary workflow data rather than on a model of your own? Send us the project.
Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.
Get the news by email
One short note each weekday. No marketing, and you can unsubscribe at any time.