# Model Evaluation

Model evaluation is the ongoing practice of measuring whether an AI system performs its intended product task acceptably across representative and risky conditions.

- Type: practice
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/model-evaluation/>)

## What it means

Model evaluation, often shortened to evals, uses datasets, scenarios, metrics, human judgments, and production evidence to assess model or system behavior. It can compare models, prompts, retrieval, tools, and end-to-end workflows. An evaluation is decision-specific: a public benchmark or one aggregate score cannot establish that a product is useful, safe, or fair in its own context.

## Why it matters

Probabilistic systems can improve one behavior while silently regressing another. Evaluations translate product expectations and risk thresholds into repeatable evidence for selection, release, monitoring, and incident response.

## When to use it

Evaluate from prototype through production whenever model, prompt, data, retrieval, tools, policy, or user distribution changes. Use offline tests for fast repeatability and online evidence for real behavior; add expert review for domains where correctness or harm requires specialist judgment.

## How it works

1. Define the product decision, task distribution, success criteria, failure severity, and comparison baseline.
2. Build versioned datasets from representative, edge, adversarial, multilingual, and high-impact cases without leakage.
3. Choose task metrics and calibrated human or automated graders, recording uncertainty and disagreements.
4. Run component and end-to-end evaluations, inspect slices and failures, and compare cost and latency alongside quality.
5. Set release thresholds, monitor production outcomes, add discovered failures to regression suites, and revisit criteria.

## A practical example

A note summarizer is evaluated on factual support, omitted decisions, action-owner accuracy, harmful content, latency, and edit distance from the final human version. The team reviews performance by meeting language and length instead of relying on one average quality score.

## Where it breaks

Test sets become stale and can be contaminated by training or repeated tuning. Human judgments vary, automated graders can share the evaluated model's biases, and offline scores may not predict user outcomes. Rare severe harms are difficult to estimate statistically.

## Common mistakes

- Selecting a model from a general benchmark without evaluating the actual workflow.
- Using an unvalidated model judge as the sole source of truth.
- Optimizing the test set until it no longer represents unseen production cases.
- Reporting average quality without failure severity, slices, cost, or latency.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/model-evaluation/#visualization-model-evaluation>)

## Sources

1. [Artificial Intelligence Risk Management Framework \(AI RMF 1.0\)](<https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10>) — National Institute of Standards and Technology · Government framework
2. [A Survey on Evaluation of Large Language Models](<https://arxiv.org/abs/2307.03109>) — arXiv · Research survey

## Related topics

- [Guardrail Metrics](<https://wiki.soroush.uk/topic/guardrail-metrics/>)
- [Data-Informed Decision Making](<https://wiki.soroush.uk/topic/data-informed-decision-making/>)
- [AI Hallucination](<https://wiki.soroush.uk/topic/hallucination/>)
