methodData, experimentation & growth

Sequential Testing

Sequential testing evaluates accumulating experiment data with stopping rules designed to preserve statistical error guarantees across repeated looks.

Download Markdown (.md)

01What it means

Sequential testing is a statistical approach in which evidence is evaluated as observations arrive and an experiment may stop according to a prespecified rule. Unlike a conventional fixed-horizon analysis, a valid sequential design accounts for repeated opportunities to declare a result, using methods such as likelihood ratios, alpha spending, confidence sequences, or always-valid p-values.[1][2]

Prespecified repeated evidence checks Continue until a valid boundary or maximum horizon is reached
  1. Predefine designHypothesis, effect, errors, maximum horizon
  2. ObserveData arrive under stable assignment
  3. Update statisticOnly at permitted analysis points
  4. Compare boundarySuccess, harm, futility, or continue
  5. Stop + reportEffect, uncertainty, stopping context, limits

If no boundary is crossed, collect more data until the maximum horizon

Ordinary repeated peeking inflates false positives; a sequential method accounts for every permitted opportunity to stop.

02Why it matters

Product teams naturally monitor experiments before a planned end date. Applying a fixed-horizon significance test every day and stopping on a favorable result inflates false positives. Sequential methods can align analysis with actual monitoring behavior, support earlier decisions when evidence is strong, and reduce exposure to harmful variants without pretending that repeated peeking is free.

03When to use it

Use sequential testing when interim monitoring or early stopping is an explicit operational requirement and the experiment platform implements a method appropriate to the metric and decision. Define practical effects, guardrails, minimum exposure, and maximum duration as well as statistical boundaries. Do not infer that any dashboard permitting live viewing automatically provides sequential validity.

04How it works

  1. Specify the hypothesis, randomization unit, primary metric, practical effect, error rates, and maximum experiment horizon.
  2. Choose and document a sequential method whose assumptions match the outcome and monitoring plan.
  3. Set decision boundaries for success, harm, or futility, plus operational minimums such as full-cycle coverage.
  4. Monitor data quality and compute the sequential statistic at permitted analysis points without changing the rule after seeing results.
  5. Stop only when a boundary or maximum horizon is reached, then report effect size, uncertainty, stopping context, guardrails, and limitations.

05A practical example

Example

A subscription service tests a cancellation-flow change and must stop quickly if support contacts rise. Its platform uses a documented sequential procedure for the primary retention metric and a harm boundary for contacts. The team monitors continuously but does not ship after the first encouraging fluctuation; it stops only when the success boundary is crossed after two complete billing cycles.

06Where it breaks

Sequential validity does not repair weak randomization, bad metrics, sample ratio mismatch, novelty, spillovers, or selective reporting. Some procedures trade flexibility for wider intervals or longer expected duration. Changing metrics, populations, variants, or stopping rules midstream can invalidate guarantees, and early statistical evidence may still lack sufficient business-cycle coverage.

Common mistakes

  • Repeatedly checking ordinary fixed-horizon p-values and calling the behavior sequential testing.
  • Stopping at statistical significance without minimum duration, guardrails, or practical-effect criteria.
  • Assuming a vendor label is sufficient without understanding the method, unit, assumptions, and reported error guarantee.

07Sources

  1. Analysis of Large Scale Web Experiments Using Sequences of EstimatorsarXiv · Research paperAnalysis of Large Scale Web Experiments Using Sequences of Estimators
  2. Statistical Analysis Methods OverviewOptimizely Support · Technical documentationStatistical Analysis Methods Overview
View saved topics

Start typing to search titles, definitions, categories, and frameworks.