Token and Tokenizer
A token is a model's processing unit, and a tokenizer is the algorithm that converts text or other input into those units and back.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/token-and-tokenizer/
01What it means
A token is an atomic unit a model processes, such as a word, subword, character, image patch, or audio segment. A tokenizer maps raw input to token identifiers and decodes output identifiers back into usable content. Token boundaries vary by model and language, so a token is not the same as a word, character, or byte.[1][2]
- 01Raw inputText, image, audio, or other modality
- 02Normalize + segmentApply tokenizer-specific rules
- 03Token unitsSubwords, characters, patches, segments
- 04Numeric IDsModel-processable representation
- 05DecodeGenerated IDs become usable content
Token boundaries vary by model, language, and modality; a token is not reliably a word, character, or byte.
02Why it matters
Tokenization affects context capacity, latency, cost, truncation, and how evenly a product supports languages and formats. Product limits expressed only in characters or pages can fail when the model actually budgets tokens.
03When to use it
Estimate and inspect tokens when designing model input limits, chunking, retrieval, streaming, pricing, or multilingual support. Users usually should see familiar units unless token counts directly affect a technical workflow or transparent billing.
04How it works
- Normalize and pre-process input according to the model's tokenizer rules.
- Segment the input into vocabulary units or modality-specific patches.
- Map each unit to a numeric identifier the model can process.
- Reserve tokens for instructions, retrieved context, conversation history, tools, and expected output.
- Decode generated identifiers and validate that truncation or unusual segmentation did not change the task.
05A practical example
A multilingual writing assistant finds that the same character limit consumes far more tokens in some languages. It changes its budget by model token counts, reserves output space, tests each supported language, and explains document limits in words rather than exposing an opaque token quota.
06Where it breaks
Token counts are model-specific and can change with a different tokenizer. Estimates may omit hidden instructions or tool schemas. Efficient tokenization does not guarantee better understanding, and aggressive truncation or chunking can remove critical relationships.
Common mistakes
- Assuming one token always equals one word or a fixed number of characters.
- Counting only user text and forgetting system instructions, history, retrieval, tools, and output.
- Applying English token estimates to every language.
- Truncating from one end without checking which information the task depends on.
07Sources
- Machine Learning GlossaryGoogle for Developers · Official technical glossaryMachine Learning Glossary
- What are tokens and how to count them?OpenAI Cookbook · Official technical guideWhat are tokens and how to count them?