# Multivariate Testing

Multivariate testing evaluates combinations of changes to multiple factors so teams can estimate component and interaction effects under a factorial design.

- Type: method
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/multivariate-testing/>)

## What it means

Multivariate testing varies two or more factors across their combinations. Unlike an A/B/n test of whole alternatives, a factorial design can estimate whether factors work independently or interact when the design and sample support those estimates.

## Why it matters

Teams often change several interface elements together and cannot tell which helped or whether a combination caused harm. A planned factorial design can yield more reusable evidence than opaque composite variants.

## When to use it

Use it when factors have credible independent or interaction hypotheses and enough traffic exists for all combinations. Prefer a simpler A/B test when one integrated experience is the actual decision.

## How it works

1. Define factors, levels, outcomes, and interactions.
2. Select a factorial or fractional-factorial design.
3. Plan sample size for main and interaction effects.
4. Run consistent combinations and check every cell.
5. Estimate effects with uncertainty before rollout.

## A practical example

A signup team tests copy and progress indicators. It finds their combination confuses first-time users, so it keeps the established pairing rather than declaring either element universally bad.

## Where it breaks

Combination counts grow quickly, producing underpowered cells. Interactions are easy to overinterpret, and a winner may not generalize.

## Common mistakes

- Calling any multi-variant test multivariate.
- Ignoring interaction effects.
- Running too many combinations for available traffic.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/multivariate-testing/#visualization-multivariate-testing>)

## Sources

1. [NIST/SEMATECH e-Handbook of Statistical Methods](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference
2. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement

## Related topics

- [A/B Testing](<https://wiki.soroush.uk/topic/a-b-testing/>)
- [Randomization](<https://wiki.soroush.uk/topic/randomization/>)
- [Sample Size](<https://wiki.soroush.uk/topic/sample-size/>)
