Context Window Compression

What is a context window?

The context window is an LLM’s working memory, measured in tokens. Everything the model can “see” — system prompt, documents, chat history, tool results, and the answer it’s writing — must fit inside it. Modern frontier models offer 200k to 1M+ token windows (hundreds of pages).

Why it matters architecturally

The context window is the budget every LLM feature spends from. RAG exists because corpora exceed it; conversation summarization exists because chats outgrow it; agent frameworks spend most of their engineering on deciding what deserves space in it. In 2026 this discipline has a name: context engineering — curating what goes in the window matters more than clever prompt wording.

Cost reality

You pay per input token on every call — the whole conversation is re-sent each turn, so a chat’s cost grows quadratically with its length. A 100k-token context at ~$3/M input tokens costs ~$0.30 per message. Prompt caching (providers discount repeated prefixes up to 90%) is the single highest-leverage cost optimization in production LLM apps.

What people get wrong

• “Long context kills RAG.” Partially. Stuffing 500 pages in works, but costs more per query and models still attend unevenly to the middle of very long inputs (“lost in the middle”). Retrieval remains cheaper and often more accurate at scale.
• Ignoring output room. Filling the window with input leaves no space for the answer.
• Treating the window as free memory. Irrelevant context doesn’t just cost money — it measurably degrades answer quality. Less, but relevant, beats more.