All news

Nvidia's Megatron turns language model size into a compute budget

A large block of identical graphite cubes on a pale stone floor, one cobalt cube on top

Nvidia trained an 8.3 billion parameter language model on 512 GPUs. The size of the next language models will be set by who can pay for the hardware.

Nvidia has trained an 8.3 billion parameter language model on 512 of its V100 GPUs and published the code. Megatron, unveiled on 13 August, is built on the transformer architecture, is the largest model of its kind trained to date and is 5.6 times the size of the full GPT-2, the model OpenAI withheld from release in February.

The size of the next language models will be set by hardware budgets rather than by new methods. Each recent jump in quality has come from a larger model trained on more text, and Megatron shows that the limit now sits in the cluster. A model of this size does not fit on one processor, so it has to be split across many chips joined by fast links, the business Nvidia moved into when it agreed to buy Mellanox in March, and only a few companies own or rent clusters of hundreds of GPUs.

That moves value toward the owners of hardware and the large clouds. Every lab that follows Nvidia's published recipe buys more of its systems. The same day Nvidia showed what such a cluster buys in time: Google's BERT-Large trained in 53 minutes on one of its supercomputers, a job that takes days on a single server. Fanaura sees here the start of a split in machine learning economics, in which a handful of groups pay to train the largest language models and everyone else rents, adapts or licenses them.

The opening for founders sits on the other side of that split. BERT and its successors are public, and a model trained once at great cost can be fine-tuned for a narrow task on a single machine in hours. Nvidia also showed BERT answering a question in 2.2 milliseconds on its T4 inference chip, fast enough to put language understanding inside live products, from contract review to translation and writing tools. The AI-first companies worth building treat pretraining as someone else's capital expense and compete on proprietary data, on evaluation built for their task and on the cost of serving each request, the division of labour behind our venture thesis.

The serious objection is that size may stop paying. If quality flattens as models grow, the race for ever larger clusters loses its point and the advantage returns to whoever has the better method. Three answers will arrive soon. OpenAI, Google or another lab will reply with a still larger model, or will not. The price of renting clusters of hundreds of GPUs in the cloud will decide how many companies can train at this scale. And the large clouds will show how quickly they turn pretrained language models into a service paid for by the request.

The largest language models are becoming industrial assets that only a few companies can afford to produce. Training and applications will run on different economics, and financial investors will find most of their returns in the second, a view we set out for long-horizon capital.

For founders

Building products on pretrained language models with your own data? Send us the project.

Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.

Apply in five minutes
Subscribe

Get the news by email

One short note each weekday. No marketing, and you can unsubscribe at any time.