# Guardrail Metrics

Guardrail metrics protect important product qualities by revealing unacceptable harm while a team optimizes a primary outcome or tests a change.

- Type: metric
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-09-21
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/guardrail-metrics/>)

## What it means

Guardrail metrics are measures selected to detect deterioration in outcomes that should not be sacrificed while pursuing a primary goal. They often cover reliability, latency, safety, privacy, user satisfaction, fairness, or another part of the journey. Teams define acceptable boundaries before reviewing results.

## Why it matters

A change can improve its target metric while damaging the broader experience. Guardrails make those tradeoffs visible and reduce the chance that local optimization ships hidden costs. They also give experiment reviews a shared basis for stopping, investigating, or escalating a decision.

## When to use it

Use guardrails in experiments, launches, operational reviews, and goal systems whenever focused optimization could create side effects. Choose risks relevant to the intervention and include data-quality checks so a healthy-looking primary metric is not trusted when telemetry is broken.

## How it works

1. Name the primary outcome and identify plausible harms to users, the business, and system health.
2. Select sensitive, interpretable measures for the most important risks and define their populations and windows.
3. Set non-inferiority thresholds, alert ranges, or decision rules before results are known.
4. Monitor primary, guardrail, diagnostic, and data-quality metrics together, including important segments.
5. Investigate breaches and document the tradeoff rather than averaging harm away with an unrelated benefit.

## A practical example

A marketplace tests richer search results to increase completed bookings. Booking conversion is the primary metric; page latency, search abandonment, cancellation rate, and accessibility errors are guardrails. Conversion rises, but mobile latency breaches its threshold, so the team improves performance and reruns the test instead of shipping immediately.

## Where it breaks

Guardrails cannot cover every possible harm, and weak thresholds can legitimize damage rather than prevent it. Multiple measures complicate statistical interpretation. A flat aggregate may conceal a severe regression for a smaller group, while slow outcomes may not mature during a short experiment.

## Common mistakes

- Choosing generic guardrails unrelated to the intervention’s plausible risks.
- Setting thresholds after seeing results.
- Treating a guardrail breach as a minor footnote without an explicit decision rule.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/guardrail-metrics/#visualization-guardrail-metrics>)

## Sources

1. [Patterns of Trustworthy Experimentation: During-Experiment Stage](<https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-during-experiment-stage/>) — Microsoft Research · Industry research guidance
2. [Patterns of Trustworthy Experimentation: Pre-Experiment Stage](<https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/>) — Microsoft Research · Industry research guidance

## Related topics

- [Counter Metrics](<https://wiki.soroush.uk/topic/counter-metrics/>)
- [Key Performance Indicator](<https://wiki.soroush.uk/topic/key-performance-indicator/>)
- [Product Analytics](<https://wiki.soroush.uk/topic/product-analytics/>)
