Google's TPU makes cheap inference a moat for the largest platforms

Google's own chip runs neural networks with 30 to 80 times more work per watt, turning the cost of each prediction into a platform advantage.
Thirty to 80 times more computation per watt than contemporary processors is what Google gets from the chip it designed to run neural networks. The first performance study of its Tensor Processing Unit, released on 5 April, also puts the chip at 15 to 30 times the speed of the server processors and graphics chips it was measured against, on the workloads it has run since 2015.
The numbers move machine learning from the research budget into the cost of goods. Training a model is a one-off expense; running it for every photo, sentence and spoken word is a bill that grows with each user. Google built the chip because of that bill: with the processors it had, people speaking to its speech recognition for three minutes a day would have forced it to double its data centers. A platform that serves predictions at a fraction of the energy of rented hardware can give away machine learning features that a smaller rival must charge for.
The chip is narrow by design. Its core is a grid of 65,536 multipliers working on 8-bit numbers, with a peak of 92 trillion operations a second, and three families of neural networks make up 95% of the inference demand in Google's data centers. Specialization of that degree pays only where demand is enormous and predictable, which is why it starts at the largest platform, the same logic that led Alphabet to build its own lidar for Waymo. The chips already serve Google Photos, Translate and the Cloud Vision API.
General-purpose chips still sell. Nvidia's data center revenue rose 145% to $830 million in the year to January, with $296 million in the final quarter. Much of that money pays for training, where flexibility matters most. Inference is where a custom chip wins, and inference is the cost that scales with users.
For founders building on rented compute, the cheapest way to serve some predictions is now to rent them from the platform that owns the chip. A start-up calling the Cloud Vision API buys image recognition priced off TPU economics; one running its own models on rented graphics chips pays the older cost per prediction. Fanaura asks every machine learning company what a prediction will cost at ten times today's users, because that decides whether margins widen with growth. Keeping a proprietary model makes sense where the model itself is why customers pay, a test our thesis applies to AI-first companies.
The comparison is dated, which is the best argument against it. The study measures the TPU against an Intel Haswell server processor and an Nvidia K80, both in service in 2015, and newer graphics chips narrow the gap while staying available to every company. Two developments will settle it: whether Google rents the chip itself to outside customers, and how Nvidia's next generation performs on inference rather than training. As long as the chip stays inside Google, its advantage is a margin no competitor can buy.
At consumer scale, machine learning has become a hardware cost problem, and the companies that design their own chips will pay the least to solve it.
Building an AI-native company? Apply in five minutes. →
More from Fanaura
Building machine learning products with a clear cost per prediction at scale? Send us the project.
Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.
Get the news by email
One short note each weekday. No marketing, and you can unsubscribe at any time.