All news

Open research groups can now build GPT-3 scale models

Many small white blocks form one large arch on an oak table, with a single cobalt keystone at the top

BLOOM matches GPT-3 in size and ships open weights in 46 languages. Model scale stops being a moat, and talent and use become the scarce assets.

BLOOM has 176 billion parameters, one more billion than GPT-3, and anyone who accepts its Responsible AI License can download the weights. The BigScience workshop, a year-long research collaboration coordinated by Hugging Face, released the model on 12 July together with its intermediate training checkpoints.

Model scale has stopped being a moat. Until now language models of this size came from a handful of corporate labs, and outsiders reached them through an API or not at all. BLOOM shows that the scarce inputs are compute and coordinated talent, and that a public grant and an open research network can supply both. That puts a ceiling on how long any company can defend an advantage built on model size alone, a test for the labs that drew the largest cheques this spring, and it adds open competition to the price of API access.

The way it was built matters as much as the result. More than 1,000 researchers from over 70 countries worked on it, and the final training run took 117 days on the Jean Zay supercomputer near Paris under a public compute grant from French research agencies. The project leaves behind a large pool of researchers with hands-on experience of training at this scale, a skill few companies have in-house. Corporate practice looks different: in May, Meta shared its 175-billion-parameter OPT model only with researchers on request, under a non-commercial license.

Languages are where openness turns into a market. BLOOM writes text in 46 natural languages and 13 programming languages, and for most of them, including Spanish, French and Arabic, it is the first model above 100 billion parameters. A small team can now build products for Latin America, Europe and the Middle East without waiting for a large lab to prioritise them, and value moves toward fine-tuning, efficient serving and proprietary data in specific languages and domains. The license deserves a reading with counsel, since it allows building on the model but restricts certain uses.

Serving is the bottleneck. The full model runs on eight 80 GB A100 GPUs, too expensive per copy for most products, so compression, distillation and inference tooling have real demand.

Size on paper is not quality in use. Independent evaluations against GPT-3 on non-English tasks will show whether the parameter count translates into results, and the price and speed of the hosted inference service Hugging Face is preparing will decide whether BLOOM is usable without owning hardware. The longer signals are whether more governments fund public compute for open models, and how many BigScience researchers go on to start companies.

A model at the scale of the best commercial system can now come from public money and open collaboration. Fanaura invests across the US, the EU and Latin America with a horizon of years, and its thesis favours research-heavy teams that turn open models into products for markets the large labs serve last. Owning a model now matters less than owning its use.

For founders

Building products on open language models for markets beyond English? Send us the project.

Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.

Apply in five minutes
Subscribe

Get the news by email

One short note each weekday. No marketing, and you can unsubscribe at any time.