Amazon's Inferentia makes the cost of running models a cloud contest

Amazon put its own inference chip into general use, claiming 40% lower cost per inference. Cloud providers now compete on what it costs to serve a model.
Up to 40% lower cost per inference is the promise behind Inf1, the cloud servers Amazon Web Services made generally available on 3 December. They run on Inferentia, the machine learning chip Amazon designed in-house and announced in November 2018, with up to 16 chips in each instance. Amazon rates Inf1 at up to three times the throughput of its G4 instances, which run on Nvidia T4 graphics processors, at that lower cost per inference.
The chip turns the cost of serving models into the ground on which clouds compete, and that moves margin. Inference, running a trained model on new data each time a user acts, is the recurring bill of machine learning, and Amazon puts it at as much as 90% of the compute cost of an application. Training happens a few times; inference grows with every customer. A cloud that owns the silicon for that workload keeps part of what used to go to the chip maker, can pass part of it on to the companies running models at scale, and depends less on a single chip supplier.
Amazon has built the plumbing to make the switch easy. A compiler called Neuron converts models trained in TensorFlow, PyTorch and MXNet to run on Inferentia, while training stays on GPU instances. The same day the company previewed Graviton2, the second generation of its own Arm-based server processor, built on a 7-nanometer process. The largest cloud provider now designs its own chips for general computing and for inference, which leaves Nvidia's data center business facing a major customer that is also building a rival product.
For an AI-first product with heavy usage, cost per inference is a gross margin line, and a 40% cut on that line changes unit economics more than most pricing decisions. The price of the cut is portability. Models compiled for a cloud provider's own chip tie a company more closely to that provider, and moving later costs engineering time. Teams that benchmark their models across chip types, keep training independent of the serving hardware and treat chip choice as part of the negotiation with their cloud get the saving without handing over the exit. Serving cost is among the first numbers Fanaura reviews in its work with founders whose products run models all day.
Launch benchmarks are not customer results, and that gap is where the argument can still fail. Inf1 has to show the same economics on customer workloads beyond the numbers Amazon published at launch. The response of the other large clouds and of Nvidia comes next, and with it an open question for independent chip start-ups: whether they can still win cloud and enterprise buyers once the largest clouds design their own silicon.
The cost of running models is set to fall faster than the cost of training them, and the cloud providers intend to capture part of that fall. The companies worth owning are those whose products earn more as inference gets cheaper, the cost curve behind our thesis.
Building an AI-native company? Apply in five minutes. →
More from Fanaura
Building an AI product where serving cost decides the margin? Send us the project.
Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.
Get the news by email
One short note each weekday. No marketing, and you can unsubscribe at any time.