Why 9B? The case for compact AI models

Model names come with a number attached: 9B, 30B, 120B, 550B-A55B. The number is the parameter count, and it decides more about a model than any other single figure: how much memory it needs, how fast it answers, what it costs to run and, to a degree, how much it can know. KTwo Crust 1 Beta has 9 billion. Here is what that means, and why compact can be the right size.

What a parameter is

A parameter is one learned number in the network: a weight in one of its matrices. Training adjusts all of them, a little at a time, until the model predicts its training data well. A 9B model has about nine billion. Everything the model knows, from grammar to facts to coding habits, is stored in that pattern of numbers.

More parameters give a model more capacity to store knowledge and represent complex patterns, which is why the largest models know the most obscure facts. But capacity isn’t quality: data, training and post-training matter just as much, and a well-trained small model can beat a poorly trained large one.

The memory math

To run a model, its weights have to sit in memory, usually a GPU’s. The size is simple arithmetic: parameters × bytes per parameter.

Those are the weights alone. A running model also needs memory for the KV cache, the stored attention state of the conversation so far, which grows with context length and with the number of conversations served at once.

The practical consequence: a 9B model fits comfortably on a single data-center GPU at full 16-bit precision (an NVIDIA H100 has 80 GB), with room left for long contexts and many parallel users. Quantized to 4 bits, it fits on a laptop GPU. A 70B model at 16 bits doesn’t fit on one H100 at all and has to be split across several.

Speed: why smaller answers faster

Text is generated one token at a time, and for each token a dense model reads every one of its weights from memory. On modern hardware that reading, not the arithmetic, is usually the bottleneck. So a rough ceiling on how fast a single conversation can go is the memory bandwidth divided by the size of the weights.

An H100 (SXM) moves about 3.35 TB a second. For a 9B model in 16-bit, that gives a ceiling of roughly 3,350 ÷ 18 ≈ 186 tokens a second for one conversation. The same sum for a 70B model gives about 24, and its weights don’t fit on one H100 in the first place. Real systems land below these ceilings and win back throughput by batching many conversations together, but the proportion holds: fewer bytes per token means faster tokens.

Cost: the same arithmetic, in dollars

Serving cost tracks the same quantities: how many GPUs a model occupies and how many tokens each one produces per hour. A compact model needs fewer GPUs per user and produces more tokens per GPU-hour, which is why small models can be cheap to offer. Crust 1 Beta is priced at $0.04 per million input tokens and $0.15 per million output tokens; see Pricing, and our guide to what per-million-token prices mean.

Reading names like 30B-A3B: total vs active parameters

Some names carry two numbers. Nemotron 3.5 Lightning 30B-A3B and Nemotron 3 Ultra 550B-A55B, both in LE PEAK, are mixture-of-experts (MoE) models. The first number is the total parameter count; the “A” number is how many parameters are active for each token. Gemma 4 26B A4B, also in LE PEAK, uses the same convention.

An MoE model splits parts of the network into many “experts”, and a router sends each token through only a few of them. So 30B-A3B holds 30 billion parameters in memory but does roughly the computation of a 3B model per token.

The trade-off: MoE models compute like small models but still need memory for all their parameters, so 550B-A55B needs the memory of a 550B model. A dense model is the same size in both senses: a dense 9B model computes with all 9 billion parameters for every token, and needs memory for exactly those.

What a compact model gives up

Being honest about size means saying what it costs:

• Long-tail knowledge. Fewer parameters store fewer rare facts. A 9B model is more likely than a 500B one to be unsure about an obscure paper or a minor historical figure.
• Headroom on the hardest reasoning. On the most demanding multi-step problems, larger models still tend to do better.
• Breadth. Capacity is shared across every language and domain the model knows.

Each has a well-known mitigation. Retrieval puts the long tail in the prompt instead of the weights. Tools such as code execution, search and calculators cover exact work. Specialization lets a smaller model match bigger ones in its own domain, which is the thinking behind KTwo Mantle 0.5, a 30B model trained on Hungarian law. And a model that keeps training can stay current without growing; see What is a self-training AI model?

When compact is the right choice

A compact model is often the better engineering choice, not a compromise:

• High-volume work: classification, extraction, tagging, routing and summarization, where the cost per call is multiplied by millions.
• Latency-sensitive products: autocomplete, voice, and agents that make many calls in a row.
• Long contexts and many users, where the memory a small model leaves free goes to the KV cache.
• Private or on-premises deployment, where the model has to fit on hardware you control.

Frequently asked questions

How much memory does a 9B model need?

About 18 GB for the weights at 16-bit precision, 9 GB at 8-bit and 4.5 GB at 4-bit, plus memory for the KV cache, which grows with context length and the number of parallel conversations.

What does 30B-A3B mean in a model name?

It marks a mixture-of-experts model with 30 billion parameters in total, of which about 3 billion are active for each token. It computes like a small model but needs the memory of a large one.

Are small language models worse than large ones?

At the same quality of training, larger models know more and handle the hardest reasoning better. But a well-trained compact model can beat larger ones on many tasks, especially in its own domain, and it is faster and cheaper to run.

How many parameters does KTwo Crust 1 Beta have?

Nine billion. It is natively multimodal and trains itself natively; the full spec is on the model page.