What a multi-armed bandit test is
A multi-armed bandit test compares two or more versions of a page and shifts traffic toward the one converting best while the test is still running. A three-headline test might start at 33% each and, a few days later, send 70% of new visitors to the leading headline. A classic A/B test keeps the split fixed until the end.
The name comes from slot machines, once called one-armed bandits. A gambler facing a row of them has to decide how many coins to spend learning which machine pays best, and how many to spend on the machine that looks best so far. A website test has the same choice, with visitors in place of coins.
How a bandit decides where traffic goes
Every bandit balances two jobs: exploration, which keeps sending some visitors to every version so the estimates stay fresh, and exploitation, which sends more visitors to the version that currently looks best. The algorithms differ in how they strike that balance:
- Epsilon-greedy sends a fixed share, often 10%, to random versions and the rest to the current leader.
- Upper confidence bound (UCB) favours versions whose results are still uncertain, so a version with little data gets another look before it is dropped.
- Thompson sampling works out each version’s probability of being the best and splits traffic in line with those probabilities. It is the most common choice in website testing tools.
- Contextual bandits also use facts about each visitor, such as device or location, to pick a version per person. That makes them closer to personalisation than to a test.
The goal behind all of them is to keep “regret” low. Regret is the conversions lost by showing weaker versions while the test is still learning.
Multi-armed bandit vs A/B testing
A bandit and an A/B test answer different questions. An A/B test asks how much better version B is than version A, and keeps the split even so the answer is as precise as possible. A bandit asks which version should get the traffic right now, and accepts a less precise answer about the losers in return for more conversions during the test.
That makes bandits a good fit for:
- Short-lived pages, such as a weekend sale banner or a holiday offer, where there is no time to run a full test and then roll out the winner.
- Headlines, hero images and promo copy with several versions, where the cost of showing a weak one is real and the long-term lesson matters less.
- Ongoing choices that get refreshed often, such as which offer to show in a sticky bar this month.
An A/B test is the better choice for redesigns, pricing changes, checkout changes and anything you want to learn from. It gives an effect size with a confidence range, keeps guardrail metrics comparable across versions and produces a result you can quote later. Our guide to A/B testing covers how to plan one.
A worked example
Here is a simple illustration with made-up numbers. A store tests three hero headlines over 30,000 visitors. Headline A converts at 2.0%, B at 3.0% and C at 2.5%, although nobody knows that at the start.
- With an even split, each headline gets 10,000 visitors: 200 + 300 + 250 = 750 orders.
- If a bandit spends the first part of the test learning and ends up giving B 60% of the traffic overall, with 20% each to A and C, the same visitors bring 120 + 540 + 150 = 810 orders.
The bandit’s 60 extra orders come at a price. A and C each got only 6,000 visitors, so the estimate of how much worse they are is wider. If the gap between versions is small, a bandit may also keep traffic spread out for a long time, because it never becomes confident enough to commit.
Pitfalls to watch for
Most problems with bandit tests come from conditions changing while the algorithm is still learning:
- Early noise. A bandit that moves traffic after the first few dozen visits can lock onto a lucky start. A warm-up period with an even split prevents this.
- Starved versions. Without a minimum share, a version that had a slow first day may never get enough traffic to recover. Keep a floor for every version.
- Changing behaviour. A version that wins on weekdays may lose at weekends. Bandits assume conversion rates stay stable, so run them across whole weeks when you can.
- Slow conversions. A bandit rewards what it can see. If most purchases happen days after the first visit, it will steer traffic using incomplete data. Pick a metric that happens within the visit, or use an A/B test.
- One metric only. A bandit optimises a single goal. A headline that lifts clicks but lowers orders can still win, so check guardrails separately.
- Switching versions. A returning visitor should see the same version every time. Good tools apply new shares to new visitors only.
Statistical significance also works differently. Because the split changes based on earlier results, standard p-values from the final numbers are biased, so a bandit result is better read as a decision about where traffic goes than as a measure of how big the effect is. Our guide to statistical significance explains what a fixed test measures instead.
Multi-armed bandit testing in Cuttl
In Cuttl’s Web Experiments, multi-armed bandit is one of three traffic options in a test’s Traffic step, next to an equal split and a custom split. It uses Thompson sampling on the test’s primary metric.
- Every test starts with an equal split. The bandit waits until each version has its minimum sample (between 50 and 200 visitors, 100 by default) before it moves any traffic.
- After that, it re-balances at most every 10 minutes. Each version keeps at least 5% of new visitors, and the rest is shared in line with each version’s chance of being the best.
- New shares apply to new visitors. Visitors already in the test keep the version they saw.
- Guardrails still run, every change in the split is logged in the test’s history, and every version has its own recordings and heatmaps.
AI assistants connected through the Cuttl MCP server can also draft a test with bandit traffic, and the launch still waits for a person to approve it.
