Model Inference
Model inference is the runtime process of applying a trained model to new input to produce a prediction, score, embedding, or generated output.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/inference/
01What it means
Inference occurs after training: a deployed model receives input and computes an output using learned parameters. For a classifier this may be a probability; for a language model it may involve repeatedly generating tokens. Model serving is the infrastructure that exposes and operates inference, while inference is the computation itself.[1][2]
- 01New inputPrompt, record, image, audio, or sensor state
- 02PreprocessTokenize, normalize, encode, or retrieve context
- 03Model executionApply learned parameters to compute a result
- 04Raw outputScore, class, embedding, tokens, or generation
- 05Post-processDecode, rank, threshold, validate, or constrain
- 06Product actionPresent, assist, automate, or request review
Inference produces a model result, not a complete decision. Preprocessing, thresholds, retrieval, tools, permissions, latency, and post-processing affect product behavior.
02Why it matters
Inference sits directly in the user experience and determines latency, availability, variable cost, privacy exposure, and output consistency. Product choices about model size, hosting, batching, caching, streaming, and fallbacks affect both quality and economics.
03When to use it
Plan inference whenever a feature calls a trained model in real time or in a batch workflow. Real-time serving fits interactive tasks; asynchronous or batch inference fits high-volume work that can wait. Deterministic computation may be preferable when exact, cheap, and predictable output is required.
04How it works
- Validate and transform incoming data into the representation expected by a pinned model version.
- Run the model on suitable compute to produce scores, vectors, or token probabilities.
- Apply decoding, thresholds, business rules, or post-processing appropriate to the task.
- Validate and route the result to the interface, workflow, fallback, or human review.
- Record privacy-safe quality and operational signals to monitor latency, cost, errors, and change.
05A practical example
A commerce app runs image-classification inference after a seller uploads a product photo. The result suggests a category but does not publish automatically; low-confidence predictions enter manual review, and a queue absorbs traffic when the serving endpoint is slow.
06Where it breaks
Inference quality is bounded by the model and input distribution. Scaling can be expensive, hardware and providers can constrain deployment, and optimizations such as quantization or aggressive caching may change quality. Logging input and output can create privacy and security risk.
Common mistakes
- Using inference and training as interchangeable terms.
- Budgeting for model access but ignoring retries, output length, review, and peak capacity.
- Optimizing average latency while ignoring tail latency and failure recovery.
- Changing model versions without end-to-end regression evaluation.
07Sources
- Machine Learning GlossaryGoogle for Developers · Official technical glossaryMachine Learning Glossary
- What is Machine Learning?Google for Developers · Official educational documentationWhat is Machine Learning?