Sample ratio mismatch: how to spot it and what to do

What sample ratio mismatch means

Sample ratio mismatch (SRM) is when the share of visitors in each version of an A/B test differs from the split you set, by more than chance explains. Set a 50/50 split, get 5,200 visitors in A and 4,800 in B, and the test has an SRM. Something is adding or losing visitors on one side, so the results can’t be trusted as they stand.

An uneven split you chose on purpose, such as 90/10 for a risky change, is fine. The check compares what you got with what you asked for, whatever that was.

Why a small gap can still be a problem

Random assignment never gives a perfect split, so the question is whether a gap is bigger than chance would make. The standard check is a chi-square goodness-of-fit test: it compares the visitor count in each version with the count the split predicts and returns a p-value.

The same percentages mean different things at different sample sizes. With a 50/50 split:

• 520 and 480 visitors (1,000 in total): p ≈ 0.21. A gap this size is common by chance.
• 5,100 and 4,900 (10,000 in total): p ≈ 0.046. Worth a look, though most tools wouldn’t flag it.
• 5,200 and 4,800 (10,000 in total): p ≈ 0.00006. Chance is a very unlikely explanation, so this is an SRM.

A 52/48 split is harmless at 1,000 visitors and a clear warning at 10,000. That is why the check uses counts and a p-value, and why a gap that looks small by eye can still matter. Many testing tools use a strict threshold such as p below 0.001, because the check runs on every test and a looser one would raise false alarms. The guide to statistical significance covers p-values in more depth.

Why SRM makes results unreliable

An SRM means the two groups are no longer comparable, which is the whole point of randomising. If 400 visitors are missing from version B, they aren’t missing at random. They are often the visitors on a slow connection, on one browser, or arriving by one route, and those visitors may convert differently from the rest.

The result can then look significant in either direction for reasons that have nothing to do with the change you are testing. Re-weighting the numbers doesn’t fix it, because you don’t know what the missing visitors would have done.

Common causes of sample ratio mismatch

Most SRMs come from one of three places: how visitors are assigned, whether they actually see their version, or how they are counted.

• Assignment. Targeting or traffic settings changed while the test was running, a visitor’s assignment isn’t kept between pages, or another test or rule sends one group elsewhere.
• Exposure. A redirect in a split URL test loses visitors who leave before the second page loads, the variant’s code fails on some browsers, or a cached page serves the old version to people assigned to the new one.
• Counting. Bots or internal traffic land unevenly, the tracking snippet loads later in one version, or a page view is counted before the visitor sees their version and dropped after.

Redirects are a common cause in split URL tests, since one version takes an extra page load and some visitors close the tab during it.

What to do when a test has an SRM

Treat an SRM as a bug report about the test, and don’t pick a winner from it. A practical order:

• Stop reading the results. Don’t ship either version on the strength of this test.
• Find when it started. If the split was fine for a week and drifted after a release or a settings change, that change is the first suspect.
• Break the split down by device, browser and channel. An SRM that sits in one browser points to a script error; one that sits in paid traffic points to a landing page or redirect.
• Watch a few recordings of each version to check that visitors see what they were assigned.
• Fix the cause, then restart the test with fresh visitors. Mixing data from before and after the fix brings the bias back.

To check a finished test from another tool, enter the visitor counts in the A/B test significance calculator, which includes a sample ratio mismatch check.

Sample ratio mismatch checks in Cuttl

In Cuttl’s Web Experiments, every running test is checked for SRM automatically. The check compares each version’s unique visitors with the split you set, using a chi-square test.

• It starts once 100 visitors have joined the test. Before that, the report says when the check will run.
• A p-value below 0.001 flags a mismatch. The report shows a warning, “Visitors aren’t split as set up”, with the p-value, and suggests checking redirects, caching and targeting.
• Custom splits are checked against their own weights, and paused or switched-off versions are left out.
• Tests using a multi-armed bandit split are not checked, because the split moves on purpose.

AI assistants connected through the Cuttl MCP server see the same flag when they read a test’s results, so a question like “can I trust this test?” gets the SRM status in the answer.

Common questions