# What is multi-armed bandit testing?

> A multi-armed bandit splits traffic within a test, sending more visitors to the version that’s ahead. How it works, and when an even split fits better.

Source: https://cuttl.io/blog/multi-armed-bandit-testing/ · Last updated: 2026-10-28

A multi-armed bandit splits traffic within a test, sending more visitors to the version that’s ahead. How it works, and when an even split fits better.

## What a multi-armed bandit is

Multi-armed bandit testing is a way of splitting traffic inside a test. The test still compares two or more versions of a page; the bandit decides how many visitors each version gets. Instead of keeping the split even until the end, it shifts new visitors toward the version converting best while the test runs.

A test of three headlines might start at 33% each and, a few days later, send 70% of new visitors to the leading headline. With an even split, each headline keeps a third of the traffic until the test ends.

The name comes from slot machines, once called one-armed bandits. A gambler facing a row of them has to decide how many coins to spend learning which machine pays best, and how many to spend on the machine that looks best so far. A website test has the same choice, with visitors in place of coins.

## How a bandit decides where traffic goes

Every bandit balances two jobs: exploration, which keeps sending some visitors to every version so the estimates stay fresh, and exploitation, which sends more visitors to the version that currently looks best. The algorithms differ in how they strike that balance:

- Epsilon-greedy sends a fixed share, often 10%, to random versions and the rest to the current leader.
- Upper confidence bound (UCB) favours versions whose results are still uncertain, so a version with little data gets another look before it is dropped.
- Thompson sampling works out each version’s probability of being the best and splits traffic in line with those probabilities. It is the most common choice in website testing tools.
- Contextual bandits also use facts about each visitor, such as device or location, to pick a version per person. That makes them closer to personalisation than to a test.

The goal behind all of them is to keep “regret” low. Regret is the conversions lost by showing weaker versions while the test is still learning.

## Multi-armed bandit vs an even A/B split

The two splits answer different questions. An even split asks how much better version B is than version A, and keeps traffic equal so the answer is as precise as possible. A bandit split asks which version should get the traffic right now, and accepts a less precise answer about the weaker versions in return for more conversions during the test.

That makes a bandit split a good fit for:

- Short-lived pages, such as a weekend sale banner or a holiday offer, where there is no time to run a full test and then roll out the winner.
- Headlines, hero images and promo copy with several versions, where the cost of showing a weak one is real and the long-term lesson matters less.
- Ongoing choices that get refreshed often, such as which offer to show in a sticky bar this month.

An even split is the better choice for redesigns, pricing changes, checkout changes and anything you want to learn from. It gives an effect size with a confidence range, keeps guardrail metrics comparable across versions and produces a result you can quote later. Our guide to A/B testing covers how to plan one.

Targeting and traffic in Web Experiments. See how Cuttl handles multi-armed bandit testing.

## How it plays out, in numbers

Here is a simple illustration with made-up numbers. A store runs an A/B test of three hero headlines over 30,000 visitors. Headline A converts at 2.0%, B at 3.0% and C at 2.5%, although nobody knows that at the start.

- With an even split, each headline gets 10,000 visitors: 200 + 300 + 250 = 750 orders.
- If a bandit split spends the first part of the test learning and ends up giving B 60% of the traffic overall, with 20% each to A and C, the same visitors bring 120 + 540 + 150 = 810 orders.

The bandit’s 60 extra orders come at a price. A and C each got only 6,000 visitors, so the estimate of how much worse they are is wider. If the gap between versions is small, a bandit may also keep traffic spread out for a long time, because it never becomes confident enough to commit.

## Pitfalls to watch for

Most problems with bandit splits come from conditions changing while the algorithm is still learning:

- Early noise. A bandit that moves traffic after the first few dozen visitors can lock onto a lucky start. A warm-up period with an even split prevents this.
- Starved versions. Without a minimum share, a version that had a slow first day may never get enough traffic to recover. Keep a floor for every version.
- Changing behaviour. A version that wins on weekdays may lose at weekends. Bandits assume conversion rates stay stable, so run them across whole weeks when you can.
- Slow conversions. A bandit rewards what it can see. If most purchases happen days after the first session, it will steer traffic using incomplete data. Pick a metric that happens within the session, or keep the split even.
- One metric only. A bandit optimises a single goal. A headline that lifts clicks but lowers orders can still win, so check guardrails separately.
- Switching versions. A returning visitor should see the same version every time. Good tools apply new shares to new visitors only.

Statistical significance also works differently. Because the split changes based on earlier results, standard p-values from the final numbers are biased, so a bandit result is better read as a decision about where traffic goes than as a measure of how big the effect is. Our guide to statistical significance explains what a test with an even split measures instead.

## Bandit traffic in Cuttl

In Cuttl’s Web Experiments, multi-armed bandit is one of three traffic splits in a test’s Traffic step, next to an equal split and a custom split. The test itself stays the same. The bandit uses Thompson sampling on the test’s primary metric.

- Every test starts with an equal split. The bandit moves no traffic until each version has at least 100 unique visitors, or the minimum set for the test. The original and each version together also need 50 conversions and 50 unique visitors without one. If the test’s minimum conversions is above 25, both counts are twice that number. For a revenue goal, only the conversions count.
- After that, it re-balances at most every 10 minutes. Each version keeps at least 5% of new visitors, and the rest is shared in line with each version’s chance of being the best. With more than 10 versions, the original included, half the traffic is shared evenly instead.
- New shares apply to new visitors. Visitors already in the test keep the version they saw.
- Guardrails still run, every change in the split is logged in the test’s history, and every version has its own recordings and heatmaps.

AI assistants connected through the Cuttl MCP server can also draft a test with a bandit split, and the launch still waits for a person to approve it.

## Common questions

### What is the difference between A/B testing and multi-armed bandit testing?

A multi-armed bandit is a way to split traffic within a test. With the standard even split, a test keeps traffic equal until the end so it can measure the difference precisely. A bandit split moves traffic toward the leading version during the test, which brings more conversions while it runs but gives a less precise read on the weaker versions.

### What does a multi-armed bandit look like in practice?

A store testing three sale banners for a weekend promotion is a typical case. The bandit starts with an even split and then sends most visitors to the banner that is converting best, while still showing the others now and then.

### What is Thompson sampling?

Thompson sampling is a Bayesian bandit method. It estimates each version’s probability of being the best and gives each version a share of traffic in line with that probability.

### When should you not use a multi-armed bandit?

Avoid bandits when you need to know how big an effect is, when conversions happen days after the first session, or when several metrics matter. Keep the split even there.

### Does a multi-armed bandit need less traffic than an even A/B split?

It usually brings more conversions during the test, but it doesn’t make a reliable answer cheaper. With small gaps between versions, a bandit can take as long as an even split to settle on a winner.

### What is a holdout group, and why keep one after you ship a test?

### What is the flicker effect in A/B testing, and how do you stop it?

### How long to run an A/B test, and how to work it out