# Token and Tokenizer

A token is a model's processing unit, and a tokenizer is the algorithm that converts text or other input into those units and back.

- Type: framework
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/token-and-tokenizer/>)

## What it means

A token is an atomic unit a model processes, such as a word, subword, character, image patch, or audio segment. A tokenizer maps raw input to token identifiers and decodes output identifiers back into usable content. Token boundaries vary by model and language, so a token is not the same as a word, character, or byte.

## Why it matters

Tokenization affects context capacity, latency, cost, truncation, and how evenly a product supports languages and formats. Product limits expressed only in characters or pages can fail when the model actually budgets tokens.

## When to use it

Estimate and inspect tokens when designing model input limits, chunking, retrieval, streaming, pricing, or multilingual support. Users usually should see familiar units unless token counts directly affect a technical workflow or transparent billing.

## How it works

1. Normalize and pre-process input according to the model's tokenizer rules.
2. Segment the input into vocabulary units or modality-specific patches.
3. Map each unit to a numeric identifier the model can process.
4. Reserve tokens for instructions, retrieved context, conversation history, tools, and expected output.
5. Decode generated identifiers and validate that truncation or unusual segmentation did not change the task.

## A practical example

A multilingual writing assistant finds that the same character limit consumes far more tokens in some languages. It changes its budget by model token counts, reserves output space, tests each supported language, and explains document limits in words rather than exposing an opaque token quota.

## Where it breaks

Token counts are model-specific and can change with a different tokenizer. Estimates may omit hidden instructions or tool schemas. Efficient tokenization does not guarantee better understanding, and aggressive truncation or chunking can remove critical relationships.

## Common mistakes

- Assuming one token always equals one word or a fixed number of characters.
- Counting only user text and forgetting system instructions, history, retrieval, tools, and output.
- Applying English token estimates to every language.
- Truncating from one end without checking which information the task depends on.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/token-and-tokenizer/#visualization-token-and-tokenizer>)

## Sources

1. [Machine Learning Glossary](<https://developers.google.com/machine-learning/glossary>) — Google for Developers · Official technical glossary
2. [What are tokens and how to count them?](<https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken>) — OpenAI Cookbook · Official technical guide

## Related topics

- [Large Language Model \(LLM\)](<https://wiki.soroush.uk/topic/large-language-model/>)
- [Context Window](<https://wiki.soroush.uk/topic/context-window/>)
- [Model Inference](<https://wiki.soroush.uk/topic/inference/>)
