Natively multimodal models, explained

A multimodal model takes in more than one kind of data, usually text and images, sometimes audio and video too, and reasons across them: read a chart and explain it, look at a screenshot and find the bug, compare two photos. Most recent assistants are multimodal in that sense. What differs is how they got there, and the difference shows up in how well they work.

There are two broad designs. In one, vision is added to a finished language model. In the other, a single model is trained on every modality together from the start. The second is what people mean by natively multimodal.

Design 1: vision added to a language model

The most common recipe starts with two models that already exist: a language model trained on text, and a vision encoder trained on images (often a CLIP-style encoder, trained to match images with their captions). A small connector maps the encoder’s output into the language model’s input space, so an image arrives as a few hundred “visual tokens” that the language model reads like words. Then the combined system is trained a little more on image-and-text data.

LLaVA (2023) is the best-known open example: a vision encoder joined to a language model by a projection layer, then tuned on instruction-following data about images. DeepMind’s Flamingo (2022) took a related route, keeping its pretrained vision and language models frozen and inserting new cross-attention layers between them.

The appeal is obvious: you reuse expensive models instead of training from scratch, and you can attach the same vision module to a newer language model later. The cost is that the two halves never learned together. The language model learned to think in text; images reach it through a translator that was trained afterwards, on far less data than either half saw on its own.

Design 2: natively multimodal

A natively multimodal model is trained on mixed data from the start. Text, images and other modalities become tokens in one shared sequence, and a single network learns from all of it during pre-training, the most data-hungry and formative stage. Researchers call this early fusion; the add-on recipe is a form of late fusion.

Meta’s Chameleon (2024) is a well-documented example: it turns images into discrete tokens and trains one model on interleaved sequences of text and image tokens, so it can read and generate both, in any order. Google likewise describes Gemini as built to be multimodal from the start, pre-trained on different modalities together rather than assembled from separate components.

Early fusion is harder. Training from scratch on every modality is expensive, mixing modalities makes training less stable, and you can’t simply swap in a better vision module later. In return you get a model whose idea of a cat, a chart or a stack trace formed from words and pixels at the same time.

How the two compare

Neither design wins every benchmark automatically; a strong add-on model beats a weak native one. But the native design removes a structural bottleneck, and its advantages show up most on tasks where the modalities really have to work together.

Where the difference shows up

When you test a multimodal model, these are the tasks that separate the designs:

• Text inside images. Screenshots, scanned documents, slides, handwritten notes. Reading small text needs fine visual detail to survive all the way to the reasoning.
• Charts and tables as images. Not just reading the numbers, but answering questions about them.
• Several images at once. “What changed between these two versions of the page?” needs the model to hold and compare images, not describe them one at a time.
• Interleaved input. A document whose text refers to figures (“as shown in Figure 3”), where the figures only make sense with the text.
• Spatial questions. Left of, above, how many, which one is bigger.

A practical test: give the model the same information twice, once as text and once as a screenshot of that text, and ask the same question. The closer the two answers, the less the image is losing on its way in.

Crust 1 Beta: multimodal from the start

KTwo Crust 1 Beta is natively multimodal. In the words of its spec: one model, trained across modalities from the start, with nothing bolted on afterwards. KTwo hasn’t yet published which modalities Crust 1 Beta handles or how its architecture brings them together; the designs above describe the field, not Crust’s internals. It is also compact, at 9B parameters (what that buys is in Why 9B?), and it trains itself natively, which we cover in What is a self-training AI model?

Crust 1 Beta is in private beta and not publicly available. To try a vision model today, LE PEAK, our free chat, includes NVIDIA’s Nemotron 3.5 Super VL, which takes images and needs no account.

Frequently asked questions

What does natively multimodal mean?

That a single model was trained on several kinds of data, such as text and images, together from the start of pre-training, instead of having a vision or audio component attached to a finished language model.

What is the difference between early and late fusion?

Early fusion combines modalities from the start, in one model trained on mixed sequences. Late fusion processes each modality with its own model and combines the results later, for example through a connector that feeds image features into a language model.

Is a natively multimodal model always better?

No. Quality depends on data, scale and training as much as on architecture. The native design removes the bottleneck between modalities, which matters most for tasks such as reading text in images or reasoning across several images.

Is KTwo Crust 1 Beta natively multimodal?

Yes. Its spec describes it as one model, trained across modalities from the start. KTwo hasn’t yet published which modalities it handles; the model page has the current spec.