frameworkEngineering & architecture

Token and Tokenizer

A token is a model's processing unit, and a tokenizer is the algorithm that converts text or other input into those units and back.

Download Markdown (.md)

01What it means

A token is an atomic unit a model processes, such as a word, subword, character, image patch, or audio segment. A tokenizer maps raw input to token identifiers and decodes output identifiers back into usable content. Token boundaries vary by model and language, so a token is not the same as a word, character, or byte.[1][2]

Model-specific input encoding Raw content becomes vocabulary units, numeric IDs, then usable output again
  1. 01Raw inputText, image, audio, or other modality
  2. 02Normalize + segmentApply tokenizer-specific rules
  3. 03Token unitsSubwords, characters, patches, segments
  4. 04Numeric IDsModel-processable representation
  5. 05DecodeGenerated IDs become usable content

Token boundaries vary by model, language, and modality; a token is not reliably a word, character, or byte.

02Why it matters

Tokenization affects context capacity, latency, cost, truncation, and how evenly a product supports languages and formats. Product limits expressed only in characters or pages can fail when the model actually budgets tokens.

03When to use it

Estimate and inspect tokens when designing model input limits, chunking, retrieval, streaming, pricing, or multilingual support. Users usually should see familiar units unless token counts directly affect a technical workflow or transparent billing.

04How it works

  1. Normalize and pre-process input according to the model's tokenizer rules.
  2. Segment the input into vocabulary units or modality-specific patches.
  3. Map each unit to a numeric identifier the model can process.
  4. Reserve tokens for instructions, retrieved context, conversation history, tools, and expected output.
  5. Decode generated identifiers and validate that truncation or unusual segmentation did not change the task.

05A practical example

Example

A multilingual writing assistant finds that the same character limit consumes far more tokens in some languages. It changes its budget by model token counts, reserves output space, tests each supported language, and explains document limits in words rather than exposing an opaque token quota.

06Where it breaks

Token counts are model-specific and can change with a different tokenizer. Estimates may omit hidden instructions or tool schemas. Efficient tokenization does not guarantee better understanding, and aggressive truncation or chunking can remove critical relationships.

Common mistakes

  • Assuming one token always equals one word or a fixed number of characters.
  • Counting only user text and forgetting system instructions, history, retrieval, tools, and output.
  • Applying English token estimates to every language.
  • Truncating from one end without checking which information the task depends on.

07Sources

  1. Machine Learning GlossaryGoogle for Developers · Official technical glossaryMachine Learning Glossary
  2. What are tokens and how to count them?OpenAI Cookbook · Official technical guideWhat are tokens and how to count them?
View saved topics

Start typing to search titles, definitions, categories, and frameworks.