Guardrail Metrics
Guardrail metrics protect important product qualities by revealing unacceptable harm while a team optimizes a primary outcome or tests a change.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/guardrail-metrics/
01What it means
Guardrail metrics are measures selected to detect deterioration in outcomes that should not be sacrificed while pursuing a primary goal. They often cover reliability, latency, safety, privacy, user satisfaction, fairness, or another part of the journey. Teams define acceptable boundaries before reviewing results.[1][2]
- Reliability + latency
- System health must not deteriorate
- Safety + privacy
- Protected outcomes and boundaries
- Satisfaction
- Experience quality elsewhere in the journey
- Fairness
- Segment harms cannot be averaged away
- Data quality
- Trustworthy assignment and measurement
- Decision rule
- Investigate or stop on a breach
Guardrail populations, windows, thresholds, and decision rules are defined before results are known and checked by important segment.
02Why it matters
A change can improve its target metric while damaging the broader experience. Guardrails make those tradeoffs visible and reduce the chance that local optimization ships hidden costs. They also give experiment reviews a shared basis for stopping, investigating, or escalating a decision.
03When to use it
Use guardrails in experiments, launches, operational reviews, and goal systems whenever focused optimization could create side effects. Choose risks relevant to the intervention and include data-quality checks so a healthy-looking primary metric is not trusted when telemetry is broken.
04How it works
- Name the primary outcome and identify plausible harms to users, the business, and system health.
- Select sensitive, interpretable measures for the most important risks and define their populations and windows.
- Set non-inferiority thresholds, alert ranges, or decision rules before results are known.
- Monitor primary, guardrail, diagnostic, and data-quality metrics together, including important segments.
- Investigate breaches and document the tradeoff rather than averaging harm away with an unrelated benefit.
05A practical example
A marketplace tests richer search results to increase completed bookings. Booking conversion is the primary metric; page latency, search abandonment, cancellation rate, and accessibility errors are guardrails. Conversion rises, but mobile latency breaches its threshold, so the team improves performance and reruns the test instead of shipping immediately.
06Where it breaks
Guardrails cannot cover every possible harm, and weak thresholds can legitimize damage rather than prevent it. Multiple measures complicate statistical interpretation. A flat aggregate may conceal a severe regression for a smaller group, while slow outcomes may not mature during a short experiment.
Common mistakes
- Choosing generic guardrails unrelated to the intervention’s plausible risks.
- Setting thresholds after seeing results.
- Treating a guardrail breach as a minor footnote without an explicit decision rule.
07Sources
- Patterns of Trustworthy Experimentation: During-Experiment StageMicrosoft Research · Industry research guidancePatterns of Trustworthy Experimentation: During-Experiment Stage
- Patterns of Trustworthy Experimentation: Pre-Experiment StageMicrosoft Research · Industry research guidancePatterns of Trustworthy Experimentation: Pre-Experiment Stage