All news

Nvidia's T4 bets AI chip demand will follow usage, not research budgets

A long grid of small graphite tiles on a pale stone floor with one cobalt tile set apart in front

Nvidia launched a 75-watt inference card and sized serving AI at $20 billion within five years. Chip demand now grows with users of AI products.

Seventy-five watts is the power budget Nvidia chose for the chip it wants in every data center server. The Tesla T4, launched on 12 September at the company's Tokyo conference, fits a standard slot, delivers 130 trillion operations a second at the 8-bit precision used for running trained models, and comes with free software to serve those models at scale. Nvidia expects the market for this work, called inference, to reach $20 billion within five years.

The launch marks a change in where demand for AI chips comes from. Training a model is a research expense, paid by teams with large budgets. Inference happens every time a customer speaks to a phone, gets a recommendation or has a photo tagged, so its volume grows with the number of users of AI products, not with the size of research labs. Nvidia built its data center business largely on training; the T4 is a bet that the larger and steadier revenue sits in serving.

Nvidia arrives from strength. Its data center revenue reached $760 million in the quarter to July, up 83% from a year earlier and the fastest growth of any of its major segments. Much of the serving work, though, still runs on the general-purpose processors already in those servers: Intel says its server chips earned more than $1 billion from AI workloads last year. That is why Nvidia frames the T4 against them and claims up to 40 times their speed. The rest of the field has seen the same opening. Five days after the T4 launch, Habana Labs, an Israeli start-up, came out of stealth with an inference processor it says handles 15,000 images a second on a standard benchmark.

Two kinds of company gain. Builders of inference chips get a market that is still forming, split by power budget and workload, and they will meet Nvidia's software, which aims to make its hardware the default before rivals ship in volume. Application companies gain more. When the cost of each prediction falls, products that were too expensive to run in real time, such as live video analysis or voice interfaces, start to clear their unit costs. Fanaura invests at that layer, where our thesis expects cheaper serving to widen margins, and founders there should keep their code portable rather than tied to one vendor's runtime.

The doubt is about who buys. The largest cloud providers already own fleets of processors, some design accelerators of their own, and a few of them could decide the inference market without paying for anyone's card. A 75-watt accelerator also adds cost to a server that runs other work. The test comes in the next few quarters: whether the major clouds offer T4 instances widely and at what hourly price, and whether Nvidia's data center growth holds once training budgets level off. Those numbers will show if serving AI becomes a merchant chip market or stays inside the clouds, a question we follow across our coverage of AI hardware.

Training made Nvidia the supplier to researchers; inference decides whether it becomes the supplier to every AI product in use.

For founders

Building AI products whose margins improve as the cost of each prediction falls? Send us the project.

Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.

Apply in five minutes
Subscribe

Get the news by email

One short note each weekday. No marketing, and you can unsubscribe at any time.