Apple's neural engine puts the cost of inference in the customer's pocket

The A11 Bionic runs neural networks on the phone at 600 billion operations a second. Everyday predictions move off the cloud bill and into the handset price.
Every iPhone Apple unveiled on 12 September carries a processor with a dedicated engine for neural networks. The A11 Bionic in the iPhone 8, the iPhone 8 Plus and the $999 iPhone X includes a dual-core neural engine that performs up to 600 billion operations per second, and it powers Face ID, which maps the owner's face with more than 30,000 infrared dots and does all of its processing on the device, not in the cloud.
The chip moves the cost of running machine learning from the data center into the customer's pocket. When a model runs on the phone, the hardware was paid for in the handset price, so an app that serves one more prediction pays nothing for the computation. That changes the unit economics of consumer machine learning products: features that would cost a cloud bill per user at scale become close to free, and data that never leaves the phone stops being a privacy liability.
The scale is what makes this a market shift rather than a feature. Apple sold 41.0 million iPhones in the June quarter alone, and every model it launches this autumn carries a neural engine. Huawei announced its own Kirin 970 at the start of the month with a dedicated neural processing unit, which it claims runs these workloads up to 25 times faster and 50 times more efficiently than the phone's main processor cores, and the Mate 10 phones built on it launch in October. Within two or three product cycles, a large share of the phones in use will carry hardware for neural networks.
Developers already have the software route in. Core ML, the framework Apple introduced in June, lets apps run trained models locally, and it reached users with iOS 11 on 19 September. Training stays on large clusters in the data center. Inference, the everyday work of applying a trained model, splits between the cloud and the device, and the device share grows with every upgrade cycle.
Founders building camera, voice or health products should plan for that split now. A model that fits on the phone can be priced per user rather than per call, and data that stays on the device makes a stronger case to cautious customers in health and finance. In Fanaura's reading of a consumer machine learning company, cost per active user after inference matters more than accuracy on a benchmark, because the first number decides whether growth makes the company richer or poorer.
The platform risk is the counterweight. Apple owns the chip, the framework and the features it builds on them, and it has not said how much of the neural engine third-party apps can reach through Core ML. Whatever Apple ships natively, from face unlock to photo sorting, no startup can sell on an iPhone. How far Core ML opens the neural engine to outside developers, and how quickly Android makers follow Huawei, will set the size of the opening.
The data center keeps the training. The margin on everyday predictions is moving to the device, and it will keep moving for several hardware cycles to come.
Building an AI-native company? Apply in five minutes. →
More from Fanaura
Building a consumer machine learning product that runs on the device? Send us the project.
Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.
Get the news by email
One short note each weekday. No marketing, and you can unsubscribe at any time.