# Multi-Armed Bandit

A multi-armed bandit repeatedly selects among options while balancing learning about uncertain choices with using choices that currently perform well.

- Type: method
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-10-06
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/multi-armed-bandit/>)

## What it means

A multi-armed bandit is a sequential decision method in which each available action, or arm, has an initially uncertain reward. The policy selects an action, observes its reward, and updates future choices. Exploration gathers evidence about alternatives; exploitation uses the current estimates. A contextual bandit also uses information available before the choice, such as task or session context. Feedback is usually observed only for the selected action.

## Why it matters

Bandits can adapt recommendations or other repeated product choices as evidence accumulates. Their objective is commonly cumulative reward during operation, while a conventional fixed-allocation A/B test primarily estimates a comparison. Adaptive allocation changes the data collected and requires appropriate evaluation methods.

## When to use it

Consider bandits when decisions recur, alternatives are acceptable to expose, rewards are measurable, and learning can occur within a useful time horizon. Prefer a planned experiment when a clear causal comparison is the main decision. Long-delayed outcomes, sparse traffic, or actions that strongly change future states can make a simple bandit formulation unsuitable.

## How it works

1. Define eligible actions, context, reward, observation window, and constraints before choosing a learning policy.
2. Select an exploration strategy that balances uncertainty with current reward estimates and limits unacceptable exposure.
3. Log the available choices, selected action, selection probability, context, and eventual reward for each decision.
4. Update the policy using observed outcomes while checking missing feedback, delayed rewards, and changing conditions.
5. Evaluate with methods suited to adaptive data, maintain a baseline or holdout where appropriate, and monitor guardrail outcomes.

## A practical example

A help center selects among three eligible article suggestions for a recurring support task. A contextual bandit uses the issue category to learn which suggestion most often resolves the request. The team tracks later repeat contacts as a guardrail and compares the policy with its existing ranking, rather than treating article clicks as confirmed resolution.

## Where it breaks

A poorly chosen reward encourages the wrong behavior. Exploration has a real opportunity cost, early noise can distort allocation, and changing preferences can invalidate old estimates. Naive averages or ordinary fixed-allocation tests can mislead on adaptive logs. Delayed and multi-step consequences may require a richer decision model.

## Common mistakes

- Calling a policy a bandit when it always chooses the current leader and never explores.
- Using clicks as reward when the intended outcome is task completion.
- Ignoring selection probabilities and then making unsupported off-policy comparisons.
- Assuming adaptive traffic allocation automatically provides a trustworthy significance test.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/multi-armed-bandit/#visualization-multi-armed-bandit>)

## Sources

1. [Contextual Bandits](<https://vowpalwabbit.org/docs/vowpal_wabbit/python/latest/tutorials/python_Contextual_bandits_and_Vowpal_Wabbit.html>) — Vowpal Wabbit · Official documentation
2. [A Contextual-Bandit Approach to Personalized News Article Recommendation](<https://arxiv.org/abs/1003.0146>) — International World Wide Web Conference · Peer-reviewed research

## Related topics

- [A/B Testing](<https://wiki.soroush.uk/topic/a-b-testing/>)
- [Recommender System](<https://wiki.soroush.uk/topic/recommender-system/>)
- [Guardrail Metrics](<https://wiki.soroush.uk/topic/guardrail-metrics/>)
