Probability Calibration
Probability calibration describes how closely a model's predicted probabilities match observed outcome frequencies across comparable predictions.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/probability-calibration/
01What it means
A predictive model is calibrated when events assigned a given probability occur at approximately that frequency over an appropriate population. For example, about 80% of cases assigned an 80% chance of success should succeed. Calibration differs from accuracy and ranking quality: a model can order cases well while systematically overstating certainty. Generated confidence statements and token probabilities are not automatically probabilities that an answer is correct.[1][2]
Conceptual reliability diagram: distance from the diagonal shows miscalibration.
02Why it matters
Product thresholds for auto-routing, review, warnings, and expected value often depend on probabilities. Poor calibration can make a seemingly cautious threshold accept too many errors or overwhelm reviewers. Reliable probability estimates still require a separate decision about the costs of each outcome.
03When to use it
Assess calibration when model probabilities influence actions, forecasts, resource allocation, or user-facing confidence. It is less central when only ordering matters. For open-ended generation, first define a checkable event and validate a confidence estimator against labeled outcomes.
04How it works
- Define the predicted event, population, time horizon, and decision that depends on its probability.
- Collect representative labeled cases separate from training and reserve independent data for final evaluation.
- Compare predicted probabilities with observed frequencies using reliability diagrams and suitable scoring measures.
- If needed, fit a calibration mapping on held-out data and evaluate it on untouched cases.
- Choose action thresholds using consequences and review capacity, then reassess calibration by segment and over time.
05A practical example
A receipt classifier routes cases above 90% confidence to automatic coding. Held-out data shows that this group is correct only 78% of the time. The team calibrates its scores, validates on another sample, and reduces automatic coverage until the error budget is met.
06Where it breaks
Calibration is a population property, not a guarantee for one prediction. Binning and small samples can hide errors, aggregate results can conceal subgroup differences, and distribution shifts can invalidate a fitted mapping. A constant base-rate forecast can be calibrated but provide little discrimination.
Common mistakes
- Presenting model confidence as a verified probability without outcome testing.
- Confusing temperature scaling for calibration with generation sampling temperature.
- Fitting and evaluating the calibration mapping on the same examples.
- Assuming global calibration holds for every user group or future population.
07Sources
- On Calibration of Modern Neural NetworksProceedings of Machine Learning Research · Peer-reviewed researchOn Calibration of Modern Neural Networks
- Probability calibrationscikit-learn · Official documentationProbability calibration