Precision and Recall
Precision measures how many predicted positives are correct, while recall measures how many actual positives a system finds.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/precision-and-recall/
01What it means
For a declared positive class, precision is TP / (TP + FP) and recall is TP / (TP + FN), where TP means true positives, FP false positives, and FN false negatives. Recall is also called sensitivity or the true positive rate. A confusion matrix counts predicted versus actual classes, making these errors visible. Accuracy includes correct negatives; F1 combines precision and recall as their harmonic mean but does not encode every business consequence.[1][2]
| Actual × predicted | Predicted positive | Predicted negative |
|---|---|---|
| Actual positive | True positive (TP) | False negative (FN) · missed positive |
| Actual negative | False positive (FP) · false alarm | True negative (TN) |
| Metric view | Precision = TP / (TP + FP) | Recall = TP / (TP + FN) |
Precision asks how many predicted positives were correct; recall asks how many actual positives were found.
02Why it matters
Moderation, routing, detection, and retrieval products face different costs for false alarms and missed cases. A high accuracy score can conceal poor detection when positives are rare. The paired measures help teams choose an operating threshold and understand the resulting review workload.
03When to use it
Use these measures when labels or relevance judgments define a meaningful positive result. Compare thresholds on representative held-out data. They do not directly measure the usefulness of an open-ended response without a defined event or rubric.
04How it works
- Define the positive class, unit of evaluation, reference labels, and consequences of each error.
- Count true positives, false positives, false negatives, and true negatives at the chosen threshold.
- Calculate precision and recall, reporting denominators and an explicit convention when a denominator is zero.
- Compare candidate thresholds against acceptable misses, false alarms, and review capacity.
- Report class and segment results alongside uncertainty, and retest when prevalence or user behavior changes.
05A practical example
In an illustrative abuse-review queue, 100 messages are flagged and 80 are confirmed abusive. The evaluation sample contains 120 abusive messages overall. Precision is 80%, recall is about 67%, and 40 abusive messages were missed. The PM evaluates whether a lower threshold catches enough additional abuse to justify the extra review load.
06Where it breaks
Results depend on label quality, the population, and the threshold. Precision changes with class prevalence even when sensitivity and specificity remain stable. Aggregate or weighted averages can hide failures on rare classes. F1 omits true negatives and may reward a balance that does not match product risk.
Common mistakes
- Reporting accuracy alone for a rare positive class.
- Treating recall as the percentage of flagged items that are correct.
- Choosing the highest F1 without examining error consequences.
- Reporting multiclass results without explaining macro, micro, or weighted averaging.
07Sources
- Classification: Accuracy, recall, precision, and related metricsGoogle for Developers · Official documentationClassification: Accuracy, recall, precision, and related metrics
- Metrics and scoring: quantifying the quality of predictionsscikit-learn · Official documentationMetrics and scoring: quantifying the quality of predictions