methodData, experimentation & growth

Multi-Armed Bandit

A multi-armed bandit repeatedly selects among options while balancing learning about uncertain choices with using choices that currently perform well.

Download Markdown (.md)

01What it means

A multi-armed bandit is a sequential decision method in which each available action, or arm, has an initially uncertain reward. The policy selects an action, observes its reward, and updates future choices. Exploration gathers evidence about alternatives; exploitation uses the current estimates. A contextual bandit also uses information available before the choice, such as task or session context. Feedback is usually observed only for the selected action.[1][2]

Adaptive exploration and exploitation Choose an arm, observe only its reward, and update future choices
  1. 01Available armsEligible actions and constraints
  2. 02Policy choosesBalance uncertainty and estimated reward
  3. 03Action deliveredLog context and selection probability
  4. 04Reward observedUsually only for the selected arm
  5. 05Policy updatesAdapt while monitoring drift and harms

Valid evaluation records available actions, selection probability, context, delayed feedback, and guardrail outcomes for every decision.

02Why it matters

Bandits can adapt recommendations or other repeated product choices as evidence accumulates. Their objective is commonly cumulative reward during operation, while a conventional fixed-allocation A/B test primarily estimates a comparison. Adaptive allocation changes the data collected and requires appropriate evaluation methods.

03When to use it

Consider bandits when decisions recur, alternatives are acceptable to expose, rewards are measurable, and learning can occur within a useful time horizon. Prefer a planned experiment when a clear causal comparison is the main decision. Long-delayed outcomes, sparse traffic, or actions that strongly change future states can make a simple bandit formulation unsuitable.

04How it works

  1. Define eligible actions, context, reward, observation window, and constraints before choosing a learning policy.
  2. Select an exploration strategy that balances uncertainty with current reward estimates and limits unacceptable exposure.
  3. Log the available choices, selected action, selection probability, context, and eventual reward for each decision.
  4. Update the policy using observed outcomes while checking missing feedback, delayed rewards, and changing conditions.
  5. Evaluate with methods suited to adaptive data, maintain a baseline or holdout where appropriate, and monitor guardrail outcomes.

05A practical example

Example

A help center selects among three eligible article suggestions for a recurring support task. A contextual bandit uses the issue category to learn which suggestion most often resolves the request. The team tracks later repeat contacts as a guardrail and compares the policy with its existing ranking, rather than treating article clicks as confirmed resolution.

06Where it breaks

A poorly chosen reward encourages the wrong behavior. Exploration has a real opportunity cost, early noise can distort allocation, and changing preferences can invalidate old estimates. Naive averages or ordinary fixed-allocation tests can mislead on adaptive logs. Delayed and multi-step consequences may require a richer decision model.

Common mistakes

  • Calling a policy a bandit when it always chooses the current leader and never explores.
  • Using clicks as reward when the intended outcome is task completion.
  • Ignoring selection probabilities and then making unsupported off-policy comparisons.
  • Assuming adaptive traffic allocation automatically provides a trustworthy significance test.

07Sources

  1. Contextual BanditsVowpal Wabbit · Official documentationContextual Bandits
  2. A Contextual-Bandit Approach to Personalized News Article RecommendationInternational World Wide Web Conference · Peer-reviewed researchA Contextual-Bandit Approach to Personalized News Article Recommendation
View saved topics

Start typing to search titles, definitions, categories, and frameworks.