How long to run an A/B test, and how to work it out

How long an A/B test should run

An A/B test should run until it reaches the sample size you planned before launch, rounded up to whole weeks, and for at least one full week. Divide the visitors the test needs by the eligible visitors you get each day. A test that needs 31,200 visitors per version on a page with 2,000 visitors a day needs 32 days, so about 5 weeks.

Most tests on busy pages take two to six weeks. A test shorter than a week misses part of the weekly cycle, and after about eight weeks the visitors in the test start to change in ways that blur the result.

Work out the duration in four steps

You need four numbers before launch: your baseline conversion rate, the minimum detectable effect, the number of versions and your daily traffic to the test pages.

• Find the baseline. Use the conversion rate of the goal on the test pages over the last few weeks, such as 5% of visitors adding to cart.
• Pick the minimum detectable effect (MDE), the smallest change worth acting on. At a 5% baseline, a 10% relative MDE means spotting a move from 5.0% to 5.5%.
• Get the sample size per version. At 95% confidence and 80% power, the example needs about 31,200 visitors per version. A sample size calculator or a test planner does this step.
• Divide by daily traffic. Two versions need 62,400 visitors in total. At 2,000 eligible visitors a day, that is 31.2 days, which rounds up to 32 days and then to 5 whole weeks.

“Eligible” means visitors who reach the test pages and fall inside the test’s traffic share. If a test runs on half of the traffic, halve the daily figure first. The guide to the minimum detectable effect shows how fast the answer grows as the MDE shrinks: halve the MDE and the test needs about four times the visitors.

Why tests run in whole weeks

Visitors behave differently on different days. Weekday and weekend visitors often arrive on different devices and from different channels. A test that stops on a Thursday counts more of some days than others, so its result leans toward those visitors.

Running whole weeks gives each day of the week the same weight in both versions. Many teams also cover one or two full business cycles, for example a monthly payday pattern for a store, so a single good or bad week can’t decide the test.

When you can stop a test

Stop a test when it reaches its planned sample size, unless the method you use is built for early stopping. Checking a fixed-horizon test every day and stopping at the first significant result raises the false positive rate well above the 5% the test was designed for, because random swings cross the line at some point.

• Fixed-horizon tests: read the result once, at the planned end.
• Sequential tests: you can check daily and stop early, because the bar for an early call is set higher to pay for the extra looks. The guide to sequential testing explains the trade-off.
• Bayesian tests: a stricter threshold before the planned end does the same job, so an early winner has to be very clear.

Stopping early for harm is different. If a version breaks checkout or clearly hurts revenue, end it and look at what went wrong.

What to do when the answer is too long

If the calculation says ten or twenty weeks, the test isn’t realistic on that page. Cookie loss, campaigns and seasonal swings build up over long runs. Most teams set a limit of four to eight weeks. When the estimate goes past it, change the plan before launch:

• Test a bolder change. A larger MDE needs far fewer visitors, and a new checkout layout is more likely to move conversions by 15% than a new button colour.
• Move the test to a busier page or a broader goal, such as clicks on Add to cart in place of completed orders.
• Test fewer versions. Each extra version adds a full sample to the total.
• Ship the change and measure it with a before and after analysis. It is weaker evidence than an A/B test, but it beats a test that never finishes.

A/B test duration in Cuttl

In Cuttl’s Web Experiments, the pre-test planner works out the duration while you set up a test, so you see it before launch.

• It reads the goal’s baseline on the test pages over the last 28 days and the pages’ recent traffic at the test’s traffic share. Once the test runs, it uses the rate observed in the test.
• It estimates visitors per version, then the days and whole weeks the test needs, and updates as you change the MDE, the power or the number of versions.
• You set the longest acceptable run, 6 weeks by default. The verdict is “A/B feasible in about N weeks”, or “Not enough traffic”, with the suggestion to test a bolder change or use a before and after analysis (Change Impact).

Cuttl uses Bayesian statistics by default and checks each test once a day. Before the planned end, a version is called better or worse only at a 99.9% chance, so checking results early won’t push a version over the line on a lucky day.

Common questions