What is a context window? AI memory limits, explained

A language model has no memory between requests. Everything it knows about your conversation, every earlier message, every attached file, the instructions it was given, has to be sent again each time and fit inside one limit: the context window. Here is what that limit counts, why it exists, what it costs, and why a model that accepts a million tokens doesn’t necessarily use them all well.

What the context window counts

The context window is the maximum number of tokens a model can process in one request, counting both what goes in and what comes out. Tokens are the pieces text is split into, often a word or part of a word; how they are cut, and why some languages need many more of them, is in What is a token?

One request typically holds:

• the system prompt, the instructions an application gives the model;
• the conversation so far, every user message and every reply;
• attachments, turned into text or image tokens;
• the reply the model is about to write.

So a model with a 128,000-token window, asked to write a 4,000-token answer, can take at most about 124,000 tokens of input. Many APIs also cap the reply separately: in LE PEAK, replies are capped at 8,192 or 4,096 tokens depending on the model, however large its window is.

Chat feels like memory, but it’s re-reading

When you send the tenth message in a chat, the application sends the model the system prompt, all nine earlier exchanges and the new message. The model reads it all from the start and writes the next reply. It doesn’t remember the earlier turns; it re-reads them.

Two consequences follow. A long conversation eventually doesn’t fit, and the application has to drop or summarize the oldest parts, which is when a model “forgets” what you said at the start. And every turn sends more input than the last, so a long thread costs more per message than a short one.

Why the limit exists

Two things grow with the length of the context.

Compute. The transformer’s attention lets every token look at every earlier one (Vaswani et al., 2017). Doubling the context roughly quadruples the attention work for processing it in full. Engineering tricks such as FlashAttention make this much faster in practice, but the growth remains.

Memory: the KV cache. To avoid recomputing the whole conversation for every new token, a model stores intermediate results, the keys and values of attention, for every token it has seen. Its size is simple arithmetic:

KV cache = 2 × layers × KV heads × head size × bytes per number × tokens

For Meta’s Llama 3.1 8B (32 layers, 8 key-value heads of 128 dimensions, 16-bit numbers), that is 128 KiB per token. At the model’s full 128,000-token context, the cache alone takes about 16 GB, roughly as much memory as the model’s 8 billion weights themselves, and that is for one conversation. A server holding many long conversations at once runs out of memory for the cache long before it runs out of compute. Why compact models leave more room for it is in Why 9B?

What a long context costs you

API prices are per token, and the whole context is input on every request. Take a document of 100,000 tokens and ask 50 questions about it in one thread: that is at least 5 million input tokens, plus the growing conversation, even though you only uploaded the document once. At KTwo Crust 1 Beta’s published price of $0.04 per million input tokens, that is $0.20; at $3 per million, a common price for large models, it is $15. Some providers discount input they have seen before (“prompt caching”), which helps exactly this pattern. Worked examples are in LLM API pricing explained.

Bigger isn’t always better

A model that accepts a long context doesn’t necessarily use all of it well.

• Lost in the middle. A 2023 study, Lost in the Middle, found that models answered best when the relevant information was at the beginning or the end of a long input, and noticeably worse when it sat in the middle.
• Advertised versus effective length. The RULER benchmark (2024) tested models on tasks harder than finding a single fact, and found that many fell well short of their advertised lengths: performance dropped long before the window was full.
• Noise. Irrelevant material in the context can distract a model just as it would a person skimming a long file.

In practice, it pays to send what matters rather than everything: the relevant sections of a document, not the whole archive. That is the idea behind retrieval-augmented generation (RAG), which searches a large collection and puts only the best-matching passages into the context.

How to make the most of a context window

• Put the instructions and the question where they’re seen. Near the start or the end of a long prompt, not buried in the middle.
• Start a new thread for a new task. A fresh context is cheaper and less distracted.
• Trim attachments. Send the pages or sections you’re asking about.
• Summarize long threads. Ask the model for a summary, then continue from it in a new thread.
• Count tokens, not words, especially in languages other than English.

Context windows in LE PEAK

Attachments in LE PEAK go again with every later message of a thread, so you can keep asking about the same file. When a thread grows past what the model can take, LE PEAK leaves out the oldest files, names them in the request instead, and the reply tells you so; see Attach PDFs, images, video and audio. TypeSafe’s Jev models take 64,000 tokens a request, or 262,000 with their long-context option.

Frequently asked questions

What is a context window in AI?

The maximum number of tokens a language model can process in one request, counting the instructions, the conversation so far, any attached files and the reply it writes.

Does the context window include the model’s answer?

Yes. The reply’s tokens share the window with the input, so a long answer leaves less room for input. Many APIs also cap the reply’s length separately.

Why does an AI chat forget what I said earlier?

Because the conversation outgrew the context window, and the application dropped or summarized the oldest messages to make room. The model re-reads the conversation on every turn and has no other memory of it.

Is a bigger context window always better?

No. Long contexts cost more memory and money, and many models use the middle of a very long input less reliably than its beginning and end. Sending only the relevant material usually gives better answers.

How many words fit in a 128,000-token context window?

For English, roughly 90,000 to 100,000 words, using OpenAI’s rule of thumb of about three-quarters of a word per token. Languages that split into more tokens fit fewer words: in our measurement, Hungarian took about twice as many tokens as English for the same sentence.