termData, experimentation & growth

Model Drift

Model drift describes changes in deployment data or relationships that can make a previously evaluated model less suitable over time.

Download Markdown (.md)

01What it means

Model drift is an operational umbrella for changing conditions around a deployed model. Data drift changes the input distribution; concept drift changes the relationship between inputs and the target outcome; prediction drift changes the output distribution. These signals are related but not equivalent. A drift alert does not prove performance has declined, and stable input distributions do not rule out concept drift.[1][2]

Different changes around a deployed model A drift signal identifies what moved—not whether performance necessarily declined
  1. 01Data driftInput distribution changes
  2. 02Concept driftInput–outcome relationship changes
  3. 03Prediction driftOutput distribution changes
  4. 04Performance changeMeasured task outcomes move
  5. 05InvestigationInstrumentation, population, labels, real-world change

Stable inputs do not rule out concept drift, and changed predictions may reflect valid population change rather than model failure.

02Why it matters

A model can pass release tests and later misroute users, misrank results, or make poor forecasts as the product and its audience change. Distinguishing data change from actual quality loss helps teams avoid both missed incidents and unnecessary retraining.

03When to use it

Monitor drift for models exposed to changing behavior, markets, languages, content, or instrumentation. Include delayed outcome checks where labels arrive later. For generative products, track changes in requests and outcomes as well as explicit model, prompt, retrieval, and tool releases.

04How it works

  1. Define a reference period or dataset, important segments, expected seasonality, and the outcomes that matter.
  2. Monitor input quality and distributions separately from prediction distributions and measured task performance.
  3. Investigate alerts for broken instrumentation, population changes, delayed labels, and real changes in input-outcome relationships.
  4. Choose a response proportionate to evidence, such as repairing data, adjusting a workflow, restricting use, or retraining.
  5. Evaluate the response on recent held-out cases and previous critical cases, then update baselines with an audit trail.

05A practical example

Example

A ticket router sees more unfamiliar vocabulary after an enterprise launch. The team finds that the new customer segment has lower routing accuracy, collects reviewed tickets from that segment, and tests an updated model. A seasonal increase in total tickets alone does not trigger retraining.

06Where it breaks

Reliable performance measures may require expensive or delayed labels. Drift detectors can generate many alerts and depend on the reference window. Feedback from previous predictions may change the data being monitored, and retraining on recent data can sacrifice important older cases.

Common mistakes

  • Treating every distribution change as evidence of model failure.
  • Calling input drift and concept drift synonyms.
  • Automatically retraining before ruling out a broken data pipeline.
  • Resetting the baseline so often that gradual degradation becomes invisible.

07Sources

  1. A survey on concept drift adaptationACM Computing Surveys · Peer-reviewed researchA survey on concept drift adaptation
  2. Model monitoring in productionMicrosoft Learn · Official documentationModel monitoring in production
View saved topics

Start typing to search titles, definitions, categories, and frameworks.