# Sequential Testing

Sequential testing evaluates accumulating experiment data with stopping rules designed to preserve statistical error guarantees across repeated looks.

- Type: method
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-09-23
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/sequential-testing/>)

## What it means

Sequential testing is a statistical approach in which evidence is evaluated as observations arrive and an experiment may stop according to a prespecified rule. Unlike a conventional fixed-horizon analysis, a valid sequential design accounts for repeated opportunities to declare a result, using methods such as likelihood ratios, alpha spending, confidence sequences, or always-valid p-values.

## Why it matters

Product teams naturally monitor experiments before a planned end date. Applying a fixed-horizon significance test every day and stopping on a favorable result inflates false positives. Sequential methods can align analysis with actual monitoring behavior, support earlier decisions when evidence is strong, and reduce exposure to harmful variants without pretending that repeated peeking is free.

## When to use it

Use sequential testing when interim monitoring or early stopping is an explicit operational requirement and the experiment platform implements a method appropriate to the metric and decision. Define practical effects, guardrails, minimum exposure, and maximum duration as well as statistical boundaries. Do not infer that any dashboard permitting live viewing automatically provides sequential validity.

## How it works

1. Specify the hypothesis, randomization unit, primary metric, practical effect, error rates, and maximum experiment horizon.
2. Choose and document a sequential method whose assumptions match the outcome and monitoring plan.
3. Set decision boundaries for success, harm, or futility, plus operational minimums such as full-cycle coverage.
4. Monitor data quality and compute the sequential statistic at permitted analysis points without changing the rule after seeing results.
5. Stop only when a boundary or maximum horizon is reached, then report effect size, uncertainty, stopping context, guardrails, and limitations.

## A practical example

A subscription service tests a cancellation-flow change and must stop quickly if support contacts rise. Its platform uses a documented sequential procedure for the primary retention metric and a harm boundary for contacts. The team monitors continuously but does not ship after the first encouraging fluctuation; it stops only when the success boundary is crossed after two complete billing cycles.

## Where it breaks

Sequential validity does not repair weak randomization, bad metrics, sample ratio mismatch, novelty, spillovers, or selective reporting. Some procedures trade flexibility for wider intervals or longer expected duration. Changing metrics, populations, variants, or stopping rules midstream can invalidate guarantees, and early statistical evidence may still lack sufficient business-cycle coverage.

## Common mistakes

- Repeatedly checking ordinary fixed-horizon p-values and calling the behavior sequential testing.
- Stopping at statistical significance without minimum duration, guardrails, or practical-effect criteria.
- Assuming a vendor label is sufficient without understanding the method, unit, assumptions, and reported error guarantee.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/sequential-testing/#visualization-sequential-testing>)

## Sources

1. [Analysis of Large Scale Web Experiments Using Sequences of Estimators](<https://arxiv.org/abs/1710.01285>) — arXiv · Research paper
2. [Statistical Analysis Methods Overview](<https://support.optimizely.com/hc/en-us/articles/39714777161229-Statistical-analysis-methods-overview>) — Optimizely Support · Technical documentation

## Related topics

- [A/B Testing](<https://wiki.soroush.uk/topic/a-b-testing/>)
- [Experimentation](<https://wiki.soroush.uk/topic/experimentation/>)
- [Sample Ratio Mismatch](<https://wiki.soroush.uk/topic/sample-ratio-mismatch/>)
