metricEngineering & architecture

Context Window

A context window is the maximum amount of tokenized input and output a model can consider within one inference request or active sequence.

Download Markdown (.md)

01What it means

The context window, often reported as a token count, bounds the material available to a model during a request. It may include system instructions, conversation history, user input, retrieved documents, tool definitions and results, and generated output. Context is temporary request-time information; it is not the same as training, persistent memory, or guaranteed attention to every included detail.[1][2]

A bounded token budget for one active model sequence Instructions, working context, evidence, and output all compete for finite space
  1. Total context windowMaximum tokenized input plus generated output
  2. Application instructionsSystem behavior, policies, and task framing
  3. Working contextCurrent request, selected conversation, and tool observations
  4. Retrieved evidenceDocuments, records, examples, and citations
  5. Output allowanceSpace reserved for reasoning and response

The window is not durable memory, guaranteed attention, or a measure of truth; long inputs can dilute relevant evidence even before the hard limit is reached.

02Why it matters

Context limits shape document size, conversation continuity, retrieval strategy, latency, and cost. Even when content technically fits, models may use information unevenly, so a larger advertised window does not automatically produce a more reliable experience.

03When to use it

Plan context explicitly for long documents, multi-turn conversations, retrieval, tools, or large outputs. Use summaries, structured state, or retrieval when keeping everything would add noise or cost; use a database for durable facts that must persist independently of a model request.

04How it works

  1. Calculate the request budget using the selected model and tokenizer.
  2. Allocate space among instructions, current input, history, retrieved evidence, tools, and output.
  3. Prioritize and structure relevant information instead of filling the window indiscriminately.
  4. Apply retrieval, summarization, or truncation with rules that preserve task-critical content.
  5. Test performance at realistic lengths and positions, then monitor over-limit and missing-context failures.

05A practical example

Example

A contract assistant cannot send every document and conversation turn indefinitely. It stores durable state outside the model, retrieves relevant clauses for each question, reserves room for citations and output, and tests whether clauses in the middle of long inputs are still used.

06Where it breaks

Nominal capacity does not measure effective recall, reasoning, or output quality. Longer input raises cost and latency and can introduce irrelevant or conflicting evidence. Summaries may lose details, while retrieval may omit the needed passage.

Common mistakes

  • Treating the maximum context size as guaranteed usable memory.
  • Filling the window with entire documents instead of selecting relevant evidence.
  • Forgetting that generated output and tool schemas may share the same budget.
  • Assuming a model will use information equally well regardless of its position.

07Sources

  1. Machine Learning Glossary: Generative AIGoogle for Developers · Official technical glossaryMachine Learning Glossary: Generative AI
  2. Lost in the Middle: How Language Models Use Long ContextsTransactions of the Association for Computational Linguistics · Peer-reviewed researchLost in the Middle: How Language Models Use Long Contexts
View saved topics

Start typing to search titles, definitions, categories, and frameworks.