frameworkResearch & behavior

Formative and Summative Evaluation

A distinction between evaluation used to improve a design and evaluation used to judge overall performance against a reference point.

Download Markdown (.md)

01What it means

Formative evaluation produces evidence for improving a product or service during design and iteration. Summative evaluation assesses overall performance at a defined point, often against an earlier version, competitor, benchmark, or release criterion. The distinction concerns the purpose and timing of evaluation, not whether the evidence is qualitative or quantitative.[1][2]

Evaluation purpose determines timing, evidence, and claim Formative evaluation improves the design; summative evaluation judges performance
Formative
Find problems, mechanisms, and opportunities while change is affordableIterative, diagnostic, and oriented toward improvement
Summative
Assess overall performance against a benchmark, target, or alternativeMore controlled, stable, and oriented toward a judgment

The same method can serve either purpose depending on the decision, timing, sampling, and analysis; summative claims require stable conditions and clear reference points.

02Why it matters

Confusing the two purposes creates mismatched studies. An exploratory usability round can reveal what to repair but cannot provide a stable benchmark, while a tightly standardized benchmark may quantify performance without explaining how to improve it. Naming the purpose clarifies methods, sample needs, measures, and decisions.

03When to use it

Use formative evaluation while shaping or revising a design and summative evaluation when a team needs a defensible comparison or overall assessment. A program may use both in sequence, and a summative study can still uncover formative observations, but the primary decision should determine the design.

04How it works

  1. State whether the decision is to improve the design or assess its overall performance.
  2. For formative work, choose methods and participants that expose causes, breakdowns, and possible changes.
  3. Iterate the design and repeat evaluation as the most consequential problems are addressed.
  4. For summative work, define stable tasks, measures, comparison points, sampling, and analysis in advance.
  5. Report conclusions within the limits of the design and carry newly discovered issues into the next formative cycle.

05A practical example

Example

A travel team runs small prototype sessions to learn why passengers miss the baggage step and revises the flow twice. Before launch, it runs a standardized study with predefined success, error, and time measures against the current product. The first studies are formative; the controlled comparison is summative.

06Where it breaks

The labels do not determine method quality, sample size, or whether a study is qualitative. A premature summative score can freeze an immature design, while endless formative rounds can postpone a necessary performance decision. Comparisons are only meaningful when tasks, participants, conditions, and measures are sufficiently compatible.

Common mistakes

  • Equating formative with qualitative and summative with quantitative.
  • Using a small diagnostic study to claim population-level improvement.
  • Running a benchmark without a stable protocol or meaningful comparison point.

07Sources

  1. Formative vs. Summative EvaluationsNielsen Norman Group · Professional guidanceFormative vs. Summative Evaluations
  2. When to Use Which User-Experience Research MethodsNielsen Norman Group · Professional guidanceWhen to Use Which User-Experience Research Methods
View saved topics

Start typing to search titles, definitions, categories, and frameworks.