A fair way to compare two versions
An A/B test, also called a split test, compares two versions of a page. Version A is what you have now. Version B has one change, such as a new headline, a shorter form or a bigger button. Visitors are split at random, half see A and half see B, and you measure which version gets more people to do what you want, like sign up, book or buy.
Because the split is random, the two groups are alike in every way except your change. That’s what makes the result fair: if B does better, the change is the reason, not the weather, an ad campaign or a busy week.
How a test works, step by step
- Start from a real problem. Heatmaps and recordings tell you what to fix, so you aren’t guessing.
- Pick one main goal. For example “completed checkout”. Decide it before you start, so you can’t move the goalposts later.
- Build version B. A visual editor lets you change text, colours and layout without code.
- Split your visitors at random and let the test run.
- Read the result once enough people have seen both versions.
- Roll out the winner to everyone, and keep an eye on it.
How long should an A/B test run?
It depends on three things: how many visitors the page gets, how often they convert today, and the smallest improvement you care about. Small improvements need a lot of visitors to tell apart from chance. Big ones show up sooner.
Two rules of thumb help. Run tests for whole weeks, because people behave differently at weekends. And don’t stop the moment a result looks good, because early leads often shrink. Cuttl’s planner works out how many weeks a test needs at your traffic before you start.
Reading the result
Most tools give you a number such as “96% chance B is better than A”. The higher it is, the more confident you can be that B really is better and the difference isn’t luck. Also check that B didn’t make anything else worse, like more errors or fewer people adding to cart. Those checks are called guardrails.
Two more checks are worth making on any result. First, the split itself: if one version received noticeably more visitors than planned, something in the setup is off (a sample ratio mismatch) and the result shouldn’t be trusted. Second, the segments: a version that wins overall can still lose on phones or for one traffic source.
Bayesian, sequential or fixed-horizon?
- Fixed-horizon tests decide the sample size up front and are read once, at the end. Checking early and stopping on a good day inflates false positives.
- Sequential tests are designed to be checked as they run, so you can stop early when the evidence is strong without that penalty.
- Bayesian results are expressed as the probability that a version is better and the likely size of the lift, which many teams find easier to act on.
None is “right” for every team. What matters is choosing the method and the main metric before launch, and sticking to them.
Other kinds of tests
- Split URL tests send part of your traffic to a completely different page, which suits full redesigns.
- Multivariate tests change several parts of a page at once and try every combination.
- Multi-page tests change every step of a funnel together, such as cart, checkout and payment.
- Multi-armed bandits send more visitors to the version that’s ahead while the test runs.
What if a page doesn’t get much traffic?
Lower-traffic pages often can’t reach a clear A/B result in a sensible time. You can still measure a change by comparing the weeks before and after it against the rest of your site. Read our guide to before-and-after checks.
A/B testing in Cuttl
Cuttl runs A/B, split URL, multivariate and multi-page tests from the same snippet as your heatmaps, with a visual editor, guardrails and three ways to read results. Every version gets its own recordings and heatmaps, so you can see why the winner won.
