All news

DALL-E and CLIP make language the way to program vision

A blank white card stands beside a glass prism that casts a soft cobalt rectangle of light on a graphite wall

OpenAI's new models learn images from captions on the open web instead of hand-labeled sets. Labeling budgets shrink and value moves to products and data.

Captions that already sit on the open web are now enough to teach a model to see. On 5 January OpenAI unveiled two models that join text and images: CLIP, which learns visual concepts from image and text pairs found online and sorts pictures into any categories named in plain words, and DALL-E, a 12-billion-parameter version of GPT-3 that draws images from a written description.

The expensive step in computer vision has been the hand-labeled dataset, slow to build and fixed to a list of concepts. CLIP replaces much of that work with captions that already exist, so a new classifier starts from category names rather than from a labeling project. Language becomes the way to program vision, and value moves away from labeling services and single-task models toward two places: the few labs that can train on web-scale data, and the companies that own the product and the customer.

The benchmark result makes the case. Without training on any of the 1.28 million labeled examples of ImageNet, CLIP matches the accuracy of the original ResNet-50 classifier on it. One pretrained model starts to serve many vision tasks, as GPT-3 did for text, and the right to use the data it learns from becomes a question for the balance sheet. OpenAI published CLIP's code and a trained model under an open license and kept DALL-E as a research demo.

For product teams the cost of trying an idea collapses. A vision feature can be tested by describing classes in words before anyone pays for a dataset, which opens room for AI-first companies in catalog tagging, content moderation, visual inspection and design tools, where categories change too often for fixed labels. Images drawn from text point to advertising, design and game art, but with DALL-E closed to builders, the near-term opportunity sits in workflows that can use an open model like CLIP today. OpenAI itself lists the limits, from counting objects to social bias, and the answer is customer data that tunes a general model to a narrow job.

The labs at the top are pulling away at the same time. Six days after the OpenAI release, Google researchers described a sparse design that scales language models to 1.6 trillion parameters at constant compute per input. Should sparse designs let the largest labs grow model size faster than their compute budgets, the gap between them and everyone else widens.

How much value reaches applications turns on two open questions. A paid API for OpenAI's image models would show whether they follow the metered path of GPT-3 and put a toll on the capability. Open replications of CLIP, and the data they are trained on, set how long any lead lasts; if they arrive fast, the advantage stays with whoever owns the customer.

General models trained on the web are spreading from text to images, and durable value sits where they meet proprietary data and paying customers. A family office investing in AI-first companies, Fanaura sets its thesis on that meeting point, and the companies and stages it backs follow from it.

For founders

Building vision products on open pretrained models with data no one else has? Send us the project.

Fanaura invests its own capital and the capital of its limited partners (LPs), usually through a separate SPV for each deal. If we see a fit, we get in touch.

Apply in five minutes
Subscribe

Get the news by email

One short note each weekday. No marketing, and you can unsubscribe at any time.