Google's Meena shows conversation quality now scales with compute

Google's 2.6-billion-parameter Meena ties better dialogue to more data and compute. Language skill now rewards the labs that can pay for scale.
Google's new conversational model scored 79% on a human-rated test of whether replies are both sensible and specific, against 86% for people and 23 points above the existing systems Google tested. Meena, presented by Google researchers on 28 January, is a neural network trained end to end to hold open conversations on any topic.
The score matters less than the curve behind it. Conversation quality tracks perplexity, a plain measure of how well a model predicts the next words, and perplexity falls as models and data grow. If quality follows scale, progress in language becomes a question of who can pay for compute and who holds the data. Value in language technology then splits in two: a handful of labs will own the largest models, and many companies will earn their margins on what they build around them.
Meena's bill shows who can play. The model has 2.6 billion parameters and learned from 341 GB of text filtered from public social media conversations, about 8.5 times the data behind OpenAI's GPT-2. Training took about 30 days on a pod of 2,048 of Google's TPU v3 cores, and a month on two thousand custom chips is a budget few companies outside the largest labs can match. Google is also keeping the result in-house for now, with no public demo and no model release, citing safety and bias risks.
Competing on model size against Google is a capital contest, not a product one. The opening sits in domain data the large labs do not have, in evaluation tied to what a customer pays for, and in products that improve each time the underlying models improve, because Meena shows that better models now arrive on a predictable path. Teams that fine-tune open models such as BERT or GPT-2 on their own data can ship faster than a lab that keeps its best work behind a review. Those are the teams Fanaura wants to meet in our talks with founders building on language models.
Scale could still turn out to be one lab's story rather than a law of the field. If other labs publish larger language and conversation models in the coming months, scale is confirmed as the main lever. Google's own review adds a test of openness, since releasing the Meena checkpoint would show how far large labs will share their work. Cloud prices settle the rest. The cost of training time on cloud TPUs and GPUs, a market where clouds now sell chips of their own, decides how many companies can afford this scale at all.
Machine learning in language is turning into an industrial process driven by chips, data and engineering budgets. The largest labs will keep pushing the frontier, and the most investable companies will sit one layer above it, the layer our thesis on AI-first companies targets. The frontier belongs to whoever can buy the next point on the curve, and the margins to those who build on it.
Building an AI-native company? Apply in five minutes. →
More from Fanaura
Building language products on your own domain data and open models? Send us the project.
Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.
Get the news by email
One short note each weekday. No marketing, and you can unsubscribe at any time.