Get started
Free tool

Statistical significance calculator for A/B tests

Enter visitors and conversions for each version to get the p-value, the lift with its range and the chance to beat control.

Control (A)

Conversion rate: 3.00%

Variant (B)

Conversion rate: 3.60%

Confidence levelHow sure you want to be before calling a winner. 95% is the most common choice.
Test typeTwo-sided checks for a difference in either direction. One-sided only checks whether the variant is better, so it reaches significance sooner but can’t detect a variant that is worse.
Advanced
Planned traffic splitUsed for the sample ratio mismatch check.

Variant B is significantly better than control

B converted at 3.60% against 3.00% for control, a +20.0% relative lift. A difference this large would be unlikely if the two versions performed the same (p = 0.018).

Conversion rateConversions divided by visitors.3.60%control 3.00%
Relative liftHow much higher or lower the variant’s rate is than control’s, as a percentage of control.+20.0%
Lift range (95%)The range the true lift probably falls in, at the confidence level you chose. A wide range means the result is uncertain.+3.2% to +39.5%
p-valueThe chance of seeing a difference at least this large if the two versions really performed the same. Lower means stronger evidence.0.018
Significant at 95%?Yes when the p-value is below 1 minus the confidence level (0.05 at 95%), after any correction for multiple variants, and the lift range doesn’t cross zero.Yes
Chance to beat controlThe Bayesian probability that the variant’s true conversion rate is higher than control’s.99.1%

Traffic split looks fine (p = 1.000)

Get startedHow Cuttl decides a test

The calculation runs in your browser. Nothing you enter is sent or stored.

A statistical significance calculator tells you whether the difference between two versions in an A/B test is bigger than chance would usually produce. Enter each version’s visitors and conversions. This one runs a two-proportion z-test for the p-value and a Bayesian model for the chance to beat control, and checks that your traffic split is sound.

How to read your result

The p-value and the chance to beat control answer different questions, so it helps to read them together with the lift range.

  • p-value: the chance of a difference at least this large if both versions really performed the same. At 95% confidence, a p-value under 0.05 counts as significant.
  • Chance to beat control: the probability that the variant’s true rate is higher than control’s. It is not the same number as 1 minus the p-value.
  • Lift range: where the true lift probably falls. “+20%, between +3% and +40%” is significant but still uncertain about the size of the win.
  • Traffic split check: if a 50/50 test ends up 52/48 on a lot of traffic, something is filtering visitors unevenly. That’s called a sample ratio mismatch, and it can bias the whole result.
Read the full guide: What is statistical significance in A/B testing?

Control converts 300 of 10,000 visitors (3.00%) and the variant converts 360 of 10,000 (3.60%). The lift is +20.0%, the p-value is 0.018, and the chance to beat control is 99.1%, so the result is significant at 95%. With the same rates on 1,000 visitors each, the p-value is 0.45 and the chance to beat control is 77%: the gap looks the same, but there’s too little data to tell it from noise.

How this calculator works

The calculator uses standard formulas and runs entirely in your browser.

1

Conversion rates

Each version’s rate is conversions divided by visitors.

2

p-value

A two-proportion z-test compares the two rates using their pooled rate. Two-sided by default; one-sided if you choose it.

3

Lift range

The range for the relative lift comes from the log of the ratio between the two rates, at the confidence level you picked.

4

Chance to beat control

Each version’s rate gets a Beta distribution starting from a flat prior (every rate equally likely before any data). The calculator then works out the exact probability that the variant’s rate is higher.

5

More than two versions

Each variant is compared with control, and the significance threshold is divided by the number of comparisons (Bonferroni correction).

6

Traffic split

A chi-square test compares the visitors each version got with the split you planned. A p-value below 0.001 is flagged as a sample ratio mismatch.

Mistakes that make a significant result wrong

A calculator can only check the numbers you give it. These are the common ways a test produces a “significant” result that doesn’t hold up:

Checking every day and stopping at the first green result

This is called peeking, and with a fixed sample size it pushes the false positive rate well above 5%. Wait for the planned sample size, or use a sequential test that’s built to be checked often.

No sample size plan

Decide the smallest lift worth detecting, the minimum detectable effect (MDE), before launch. At a 3% conversion rate, detecting a 10% relative lift at 95% confidence and 80% power needs about 53,000 visitors per variant.

Too many metrics or variants

Each extra comparison is another chance of a false positive. Pick one primary metric up front.

Stopping mid-week

Run tests for whole weeks so weekday and weekend visitors are both counted.

Small changes on small sites

If a test would take months, a before-and-after comparison may be the better tool.

Measuring a change without an A/B test

Get these checks on every live test

Cuttl’s Web Experiments runs these checks for you on live tests, so you don’t need to copy numbers into a calculator.

  • Bayesian statistics by default, with sequential and fixed-horizon tests available
  • A pre-test planner that estimates how many weeks a test needs, with the minimum detectable effect set before launch
  • Peeking warnings, false-positive risk and sample ratio mismatch checks on every result
  • A launch decision with expected lift and chance to beat control, plus each variant’s own heatmaps and recordings
See test statistics in Web Experiments

Significance calculator questions

Enter each version’s visitors and conversions above. If the p-value is below 1 minus your confidence level (0.05 at 95%), the difference is statistically significant.

95% is the most common choice. Teams sometimes use 90% for low-risk changes and 99% when a wrong decision would be costly to undo.

No. The p-value describes how surprising the data would be if there were no difference. The chance that B is better is a Bayesian figure, which this calculator shows separately as the chance to beat control.

Use two-sided unless you decided before the test that you only care whether the variant is better. A one-sided test reaches significance sooner, but it can’t tell you that a variant is worse.

Yes, up to three variants against control. The calculator applies a Bonferroni correction, so each variant needs a lower p-value to count as significant.

It’s when visitors aren’t split between versions the way the test planned, for example 52/48 instead of 50/50 on a lot of traffic. It usually means something is dropping visitors from one version, and it can bias the result.

It depends on your conversion rate and the smallest lift you want to detect. At a 3% conversion rate, detecting a 10% relative lift takes about 53,000 visitors per variant at 95% confidence and 80% power.

Run A/B tests with the statistics built in

Cuttl plans the sample size, checks the traffic split and shows the chance to beat control on every test, alongside heatmaps and recordings for each variant.

Get started