# Statistical significance calculator for A/B tests > Free A/B test significance calculator. Enter visitors and conversions to get the p-value, lift with a confidence interval and the chance to beat control. Source: https://cuttl.io/tools/ab-test-significance-calculator/ · Last updated: 2026-11-02 Enter visitors and conversions for each version to get the p-value, the lift with its range and the chance to beat control. Variant B is significantly better than control Traffic split looks fine (p = 1.000) The calculation runs in your browser. Nothing you enter is sent or stored. A statistical significance calculator tells you whether the difference between two versions in an A/B test is bigger than chance would usually produce. Enter each version’s visitors and conversions. This one runs a two-proportion z-test for the p-value and a Bayesian model for the chance to beat control, and checks that your traffic split is sound. ## How to read your result. The p-value and the chance to beat control answer different questions, so it helps to read them together with the lift range. p-value: the chance of a difference at least this large if both versions really performed the same. At 95% confidence, a p-value under 0.05 counts as significant. Chance to beat control: the probability that the variant’s true rate is higher than control’s. It is not the same number as 1 minus the p-value. Lift range: where the true lift probably falls. “+20%, between +3% and +40%” is significant but still uncertain about the size of the win. Traffic split check: if a 50/50 test ends up 52/48 on a lot of traffic, something is filtering visitors unevenly. That’s called a sample ratio mismatch, and it can bias the whole result. Control converts 300 of 10,000 visitors (3.00%) and the variant converts 360 of 10,000 (3.60%). The lift is +20.0%, the p-value is 0.018, and the chance to beat control is 99.1%, so the result is significant at 95%. With the same rates on 1,000 visitors each, the p-value is 0.45 and the chance to beat control is 77%: the gap looks the same, but there’s too little data to tell it from noise. ## How this calculator works. The calculator uses standard formulas and runs entirely in your browser. ### Conversion rates Each version’s rate is conversions divided by visitors. ### p-value A two-proportion z-test compares the two rates using their pooled rate. Two-sided by default; one-sided if you choose it. ### Lift range The range for the relative lift comes from the log of the ratio between the two rates, at the confidence level you picked. ### Chance to beat control Each version’s rate gets a Beta distribution starting from a flat prior (every rate equally likely before any data). The calculator then works out the exact probability that the variant’s rate is higher. ### More than two versions Each variant is compared with control, and the significance threshold is divided by the number of comparisons (Bonferroni correction). ### Traffic split A chi-square test compares the visitors each version got with the split you planned. A p-value below 0.001 is flagged as a sample ratio mismatch. ## Mistakes that make a significant result wrong. A calculator can only check the numbers you give it. These are the common ways a test produces a “significant” result that doesn’t hold up: ### Checking every day and stopping at the first green result This is called peeking, and with a fixed sample size it pushes the false positive rate well above 5%. Wait for the planned sample size, or use a sequential test that’s built to be checked often. ### No sample size plan Decide the smallest lift worth detecting, the minimum detectable effect (MDE), before launch. At a 3% conversion rate, detecting a 10% relative lift at 95% confidence and 80% power needs about 53,000 visitors per variant. ### Too many metrics or variants Each extra comparison is another chance of a false positive. Pick one primary metric up front. ### Stopping mid-week Run tests for whole weeks so weekday and weekend visitors are both counted. ### Small changes on small sites If a test would take months, a before-and-after comparison may be the better tool. ## Get these checks on every live test. Cuttl’s Web Experiments runs these checks for you on live tests, so you don’t need to copy numbers into a calculator. - Bayesian statistics by default, with sequential and fixed-horizon tests available - A pre-test planner that estimates how many weeks a test needs, with the minimum detectable effect set before launch - Peeking warnings, false-positive risk and sample ratio mismatch checks on every result - A launch decision with expected lift and chance to beat control, plus each variant’s own heatmaps and recordings ## Significance calculator questions ### How do I know if my A/B test is statistically significant? Enter each version’s visitors and conversions above. If the p-value is below 1 minus your confidence level (0.05 at 95%), the difference is statistically significant. ### What confidence level should I use? 95% is the most common choice. Teams sometimes use 90% for low-risk changes and 99% when a wrong decision would be costly to undo. ### Is a p-value of 0.03 a 97% chance that B is better? No. The p-value describes how surprising the data would be if there were no difference. The chance that B is better is a Bayesian figure, which this calculator shows separately as the chance to beat control. ### Should I use a one-sided or two-sided test? Use two-sided unless you decided before the test that you only care whether the variant is better. A one-sided test reaches significance sooner, but it can’t tell you that a variant is worse. ### Can I compare more than one variant? Yes, up to three variants against control. The calculator applies a Bonferroni correction, so each variant needs a lower p-value to count as significant. ### What is a sample ratio mismatch? It’s when visitors aren’t split between versions the way the test planned, such as 52/48 instead of 50/50 on a lot of traffic. It usually means something is dropping visitors from one version, and it can bias the result. ### How many visitors do I need? It depends on your conversion rate and the smallest lift you want to detect. At a 3% conversion rate, detecting a 10% relative lift takes about 53,000 visitors per variant at 95% confidence and 80% power. Work it out for your page with the A/B test sample size calculator. ## Run A/B tests with the statistics built in. Cuttl plans the sample size, checks the traffic split and shows the chance to beat control on every test, alongside heatmaps and recordings for each variant.