All news

Megatron-Turing shows the largest language models are now a compute race

Hundreds of small graphite-blue cubes packed into one large block, with a single cobalt cube resting alone in front

Microsoft and Nvidia trained a 530-billion-parameter model on a 560-server supercomputer. The model layer now belongs to owners of compute.

A single copy of Megatron-Turing NLG spans 280 GPUs. Microsoft and Nvidia unveiled the language model on 11 October with 530 billion parameters, three times the size of the largest existing model of its type, OpenAI's GPT-3 with 175 billion, and call it the largest and most powerful monolithic transformer language model trained to date.

That hardware bill is the real news. The largest language models are now built where the largest pools of GPUs sit, and the model layer is concentrating in corporations that own compute. For everyone else, value shifts to what is built on top: proprietary data, fine-tuning, workflow and distribution.

The training run shows the size of the entry ticket. The model was trained on Nvidia's Selene supercomputer, built from 560 DGX A100 servers with eight A100 GPUs each, on 270 billion tokens of text from a curated mix of public sources. Training at this scale needs thousands of top-end accelerators linked by fast networks, which few companies outside the biggest clouds and chip makers can finance. The result set new marks on several reading and reasoning benchmarks, and the teams state that the model picks up stereotypes and biases from its training data.

Funding startups to race on parameter count makes little sense against that arithmetic. Fanaura, a family office investing in AI-first companies, backs teams that turn access to large models into products customers pay for, and those teams should assume the largest models will be rented, not owned. Their products need a model that can be swapped as larger and cheaper ones arrive, with scarce capital spent on proprietary data, evaluation and the customer workflow.

The release itself lowers costs one level down. Megatron-Turing is a research result built on the open-source Megatron-LM and DeepSpeed training libraries, and those libraries cut the cost of training mid-sized models on a company's own data. That suits narrow domains, where a smaller model trained on the right text beats a general giant. Bias is now a stated property of these systems, so testing and mitigation belong in the product before enterprise buyers ask for them.

The case for size is not settled. This model was trained on fewer tokens than earlier large models, which leaves open whether more parameters keep paying off against their cost or whether more and better training data proves the cheaper lever. Whether Microsoft offers models of this scale to customers through its cloud, and at what price, decides how cheaply builders can rent them. The number of other groups that announce models above 100 billion parameters over the coming year will show how many balance sheets join the race.

The parameter race is a contest between balance sheets, and it is not ours to fund. We look for products that grow stronger as models improve and back founders building on rented models, where each new and larger model lifts the product instead of replacing it.

For founders

Building products that get better each time a larger model arrives? Send us the project.

Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.

Apply in five minutes
Subscribe

Get the news by email

One short note each weekday. No marketing, and you can unsubscribe at any time.