# Model Inference

Model inference is the runtime process of applying a trained model to new input to produce a prediction, score, embedding, or generated output.

- Type: term
- Domain: Engineering & architecture
- Level: Foundational
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/inference/>)

## What it means

Inference occurs after training: a deployed model receives input and computes an output using learned parameters. For a classifier this may be a probability; for a language model it may involve repeatedly generating tokens. Model serving is the infrastructure that exposes and operates inference, while inference is the computation itself.

## Why it matters

Inference sits directly in the user experience and determines latency, availability, variable cost, privacy exposure, and output consistency. Product choices about model size, hosting, batching, caching, streaming, and fallbacks affect both quality and economics.

## When to use it

Plan inference whenever a feature calls a trained model in real time or in a batch workflow. Real-time serving fits interactive tasks; asynchronous or batch inference fits high-volume work that can wait. Deterministic computation may be preferable when exact, cheap, and predictable output is required.

## How it works

1. Validate and transform incoming data into the representation expected by a pinned model version.
2. Run the model on suitable compute to produce scores, vectors, or token probabilities.
3. Apply decoding, thresholds, business rules, or post-processing appropriate to the task.
4. Validate and route the result to the interface, workflow, fallback, or human review.
5. Record privacy-safe quality and operational signals to monitor latency, cost, errors, and change.

## A practical example

A commerce app runs image-classification inference after a seller uploads a product photo. The result suggests a category but does not publish automatically; low-confidence predictions enter manual review, and a queue absorbs traffic when the serving endpoint is slow.

## Where it breaks

Inference quality is bounded by the model and input distribution. Scaling can be expensive, hardware and providers can constrain deployment, and optimizations such as quantization or aggressive caching may change quality. Logging input and output can create privacy and security risk.

## Common mistakes

- Using inference and training as interchangeable terms.
- Budgeting for model access but ignoring retries, output length, review, and peak capacity.
- Optimizing average latency while ignoring tail latency and failure recovery.
- Changing model versions without end-to-end regression evaluation.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/inference/#visualization-inference>)

## Sources

1. [Machine Learning Glossary](<https://developers.google.com/machine-learning/glossary>) — Google for Developers · Official technical glossary
2. [What is Machine Learning?](<https://developers.google.com/machine-learning/intro-to-ml/what-is-ml>) — Google for Developers · Official educational documentation

## Related topics

- [Machine Learning \(ML\)](<https://wiki.soroush.uk/topic/machine-learning/>)
- [Large Language Model \(LLM\)](<https://wiki.soroush.uk/topic/large-language-model/>)
- [Non-Functional Requirements](<https://wiki.soroush.uk/topic/non-functional-requirements/>)
