# Model Drift

Model drift describes changes in deployment data or relationships that can make a previously evaluated model less suitable over time.

- Type: term
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/model-drift/>)

## What it means

Model drift is an operational umbrella for changing conditions around a deployed model. Data drift changes the input distribution; concept drift changes the relationship between inputs and the target outcome; prediction drift changes the output distribution. These signals are related but not equivalent. A drift alert does not prove performance has declined, and stable input distributions do not rule out concept drift.

## Why it matters

A model can pass release tests and later misroute users, misrank results, or make poor forecasts as the product and its audience change. Distinguishing data change from actual quality loss helps teams avoid both missed incidents and unnecessary retraining.

## When to use it

Monitor drift for models exposed to changing behavior, markets, languages, content, or instrumentation. Include delayed outcome checks where labels arrive later. For generative products, track changes in requests and outcomes as well as explicit model, prompt, retrieval, and tool releases.

## How it works

1. Define a reference period or dataset, important segments, expected seasonality, and the outcomes that matter.
2. Monitor input quality and distributions separately from prediction distributions and measured task performance.
3. Investigate alerts for broken instrumentation, population changes, delayed labels, and real changes in input-outcome relationships.
4. Choose a response proportionate to evidence, such as repairing data, adjusting a workflow, restricting use, or retraining.
5. Evaluate the response on recent held-out cases and previous critical cases, then update baselines with an audit trail.

## A practical example

A ticket router sees more unfamiliar vocabulary after an enterprise launch. The team finds that the new customer segment has lower routing accuracy, collects reviewed tickets from that segment, and tests an updated model. A seasonal increase in total tickets alone does not trigger retraining.

## Where it breaks

Reliable performance measures may require expensive or delayed labels. Drift detectors can generate many alerts and depend on the reference window. Feedback from previous predictions may change the data being monitored, and retraining on recent data can sacrifice important older cases.

## Common mistakes

- Treating every distribution change as evidence of model failure.
- Calling input drift and concept drift synonyms.
- Automatically retraining before ruling out a broken data pipeline.
- Resetting the baseline so often that gradual degradation becomes invisible.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/model-drift/#visualization-model-drift>)

## Sources

1. [A survey on concept drift adaptation](<https://doi.org/10.1145/2523813>) — ACM Computing Surveys · Peer-reviewed research
2. [Model monitoring in production](<https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2>) — Microsoft Learn · Official documentation

## Related topics

- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
- [Machine Learning \(ML\)](<https://wiki.soroush.uk/topic/machine-learning/>)
- [Observability](<https://wiki.soroush.uk/topic/observability/>)
