# Context Window

A context window is the maximum amount of tokenized input and output a model can consider within one inference request or active sequence.

- Type: metric
- Domain: Engineering & architecture
- Level: Foundational
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/context-window/>)

## What it means

The context window, often reported as a token count, bounds the material available to a model during a request. It may include system instructions, conversation history, user input, retrieved documents, tool definitions and results, and generated output. Context is temporary request-time information; it is not the same as training, persistent memory, or guaranteed attention to every included detail.

## Why it matters

Context limits shape document size, conversation continuity, retrieval strategy, latency, and cost. Even when content technically fits, models may use information unevenly, so a larger advertised window does not automatically produce a more reliable experience.

## When to use it

Plan context explicitly for long documents, multi-turn conversations, retrieval, tools, or large outputs. Use summaries, structured state, or retrieval when keeping everything would add noise or cost; use a database for durable facts that must persist independently of a model request.

## How it works

1. Calculate the request budget using the selected model and tokenizer.
2. Allocate space among instructions, current input, history, retrieved evidence, tools, and output.
3. Prioritize and structure relevant information instead of filling the window indiscriminately.
4. Apply retrieval, summarization, or truncation with rules that preserve task-critical content.
5. Test performance at realistic lengths and positions, then monitor over-limit and missing-context failures.

## A practical example

A contract assistant cannot send every document and conversation turn indefinitely. It stores durable state outside the model, retrieves relevant clauses for each question, reserves room for citations and output, and tests whether clauses in the middle of long inputs are still used.

## Where it breaks

Nominal capacity does not measure effective recall, reasoning, or output quality. Longer input raises cost and latency and can introduce irrelevant or conflicting evidence. Summaries may lose details, while retrieval may omit the needed passage.

## Common mistakes

- Treating the maximum context size as guaranteed usable memory.
- Filling the window with entire documents instead of selecting relevant evidence.
- Forgetting that generated output and tool schemas may share the same budget.
- Assuming a model will use information equally well regardless of its position.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/context-window/#visualization-context-window>)

## Sources

1. [Machine Learning Glossary: Generative AI](<https://developers.google.com/machine-learning/glossary/generative>) — Google for Developers · Official technical glossary
2. [Lost in the Middle: How Language Models Use Long Contexts](<https://arxiv.org/abs/2307.03172>) — Transactions of the Association for Computational Linguistics · Peer-reviewed research

## Related topics

- [Token and Tokenizer](<https://wiki.soroush.uk/topic/token-and-tokenizer/>)
- [Retrieval-Augmented Generation \(RAG\)](<https://wiki.soroush.uk/topic/retrieval-augmented-generation/>)
- [Large Language Model \(LLM\)](<https://wiki.soroush.uk/topic/large-language-model/>)
