# Data Leakage in Machine Learning

Data leakage occurs when model development or evaluation uses information that would not legitimately be available for the intended prediction.

- Type: term
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-10-06
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/data-leakage-in-machine-learning/>)

## What it means

Data leakage makes an evaluation easier than the real task by allowing inappropriate information into features, training, preprocessing, or model selection. Examples include future events used to predict an earlier outcome and near-duplicate records crossing training and test boundaries. Test contamination is a related case where evaluation material influences development or training. Here, leakage concerns invalid evidence, rather than the security meaning of unauthorized data disclosure.

## Why it matters

Inflated offline scores can lead a team to release a model that fails with new customers or future events. The product decision determines the valid information boundary: predicting another event for a known customer differs from predicting for an entirely new customer.

## When to use it

Audit leakage before trusting model comparisons, buying benchmark claims, or releasing a predictive feature. It also matters when repeatedly editing prompts against a supposedly independent test set. Realistic held-out evaluation remains necessary even if no obvious leakage is found.

## How it works

1. Specify the prediction time, target population, and information actually available when the product must act.
2. Split examples by time, customer, document family, or another relevant dependency before fitting transformations.
3. Fit preprocessing and feature selection only on the training portion, keeping evaluation data outside these steps.
4. Inspect duplicates, derived labels, future information, and repeated tuning against test results.
5. Re-evaluate on an untouched, deployment-relevant sample and document any remaining uncertainty about training exposure.

## A practical example

A team predicts which support tickets will escalate using exported records. A feature records the final assigned team, which changes after escalation. Removing it and evaluating on later tickets produces a lower but more credible score. The team postpones automatic routing and collects features available at ticket creation.

## Where it breaks

Leakage can be subtle when records are correlated or data provenance is incomplete. Foundation-model training exposure may be unknown. A clean split does not eliminate sampling bias, distribution shift, poor labels, or a mismatch between the prediction target and customer value.

## Common mistakes

- Randomly splitting related records when deployment requires generalization to new entities.
- Fitting normalization or feature selection on the full dataset.
- Using outcome-derived fields that appear only after the decision.
- Calling a repeatedly optimized test set an independent final evaluation.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/data-leakage-in-machine-learning/#visualization-data-leakage-in-machine-learning>)

## Sources

1. [Common pitfalls and recommended practices](<https://scikit-learn.org/stable/common_pitfalls.html>) — scikit-learn · Official documentation
2. [Leakage and the Reproducibility Crisis in ML-based Science](<https://arxiv.org/abs/2207.07048>) — Patterns · Peer-reviewed research

## Related topics

- [Machine Learning \(ML\)](<https://wiki.soroush.uk/topic/machine-learning/>)
- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
- [Dataset Card](<https://wiki.soroush.uk/topic/dataset-card/>)
