# What is sequential testing? > Sequential testing lets you check an A/B test while it runs and stop early without inflating false positives. Here is how it works and what it costs. Source: https://cuttl.io/blog/what-is-sequential-testing/ · Last updated: 2026-10-18 Sequential testing lets you check an A/B test while it runs and stop early without inflating false positives. Here is how it works and what it costs. ## What sequential testing is Sequential testing is a way of analysing an A/B test while data is still coming in, with statistical rules that keep the false-positive rate at the level you chose, usually 5%. A fixed-sample test is only valid if you read it once, at the planned sample size. A sequential test stays valid however often you look, so you can stop as soon as the evidence is strong enough. ## Why checking a normal A/B test early goes wrong A standard significance test assumes one look at the data. Every extra look gives chance another opportunity to cross the 5% line, so the real false-positive rate climbs. The textbook figure, from clinical-trial statistics, is that two equally spaced looks at a 5% level push the real false-positive rate to about 8%, and five looks push it to about 14%. Checking a dashboard every day for three weeks does far more damage. The result is a familiar pattern. A test shows “95% significant” on day four, the team ships the variant, and the lift never shows up in the numbers. Early gaps are mostly noise, and stopping on the first good-looking one selects for noise. Our guide to statistical significance explains what the 95% figure does and does not promise. ## How sequential testing keeps the error rate under control Sequential methods spend the allowed error gradually. Instead of one 5% threshold at the end, the test uses a stricter bar early on and relaxes it as evidence builds up, so the total chance of a false positive across all looks stays at 5%. - Decision boundaries. The test sets an efficacy boundary (stop, the variant is better) and often a futility boundary (stop, there is unlikely to be an effect). - Error spending. The 5% is shared out across the looks, so an early look can only stop the test with very strong evidence. - A running statistic. Each new batch of visitors updates the statistic, which is compared with the boundary again. An early stop needs a large gap. A late stop needs only a modest one. That asymmetry is what lets a clearly better or clearly worse variant end the test in days while a marginal one runs to the end. Statistics in Web Experiments. See how Cuttl handles sequential testing. ## Two families: group sequential and always valid Most tools use one of two designs, and they suit different data: - Group sequential tests plan a fixed number of looks, such as once a day for four weeks, and set a boundary for each. They have the most statistical power when you can estimate the maximum sample size in advance, and they suit daily batch reporting. - Always-valid tests, such as the mixture sequential probability ratio test (mSPRT), produce a p-value or a confidence sequence that stays valid at every moment. You do not need a planned number of looks or an exact sample size, which makes them a better fit for streaming data and for teams that check results whenever they like. The price of that flexibility is power. In published comparisons, always-valid methods need somewhat more traffic than a group sequential design that has the right sample size, and both need more than a single fixed-sample test that is read exactly once at the planned end. ## What sequential testing costs you Sequential testing trades a little precision for the freedom to look. The trade-offs are worth knowing before you switch: - Wider early intervals. A confidence sequence is wider than a normal confidence interval at the same sample size, so early estimates of the lift are loose. - More traffic for small effects. If the real lift is small, a sequential test may need more visitors than a fixed test read once at the end. - Overstated lifts on early stops. A test that stops because it crossed the boundary tends to report a lift that is larger than the true one. Treat the size of an early win as an upper estimate. - Not a cure for bad setup. A sample ratio mismatch, a changing audience or a promotion that runs mid-test will mislead a sequential test just as they mislead any other. ## When to use sequential testing Sequential testing fits when the cost of waiting or of a bad variant is high and results are checked often: - High-stakes changes, such as pricing or checkout, where a clearly harmful variant should stop quickly. - Teams that look at results daily, who would otherwise be tempted to stop early by hand. - Tests on pages with enough traffic that a strong effect shows up within days. A fixed-horizon test is still a good choice for small optimisations on a page with steady traffic, where the plan is to wait for the planned sample and read the result once. Bayesian methods are a third option, and they handle repeated looks differently, as the guide to what A/B testing is outlines. ## Sequential testing in Cuttl In Cuttl’s Web Experiments, the statistics step of a test offers three models: Bayesian (the default), Sequential and fixed-horizon frequentist. Sequential uses always-valid p-values based on a mixture sequential probability ratio test. - The smallest effect worth detecting, the minimum detectable effect (MDE), is set before launch, and the test uses it to tune the statistic. - Cuttl checks daily and keeps the lowest always-valid p-value seen so far, so the result never goes back to “not significant” after a lucky day. - Lift ranges come from a confidence sequence, which stays valid at every look. When several variants are compared with the control, an optional Bonferroni correction splits the error across them. - The results page shows progress against the planned sample and says when a sequential test is safe to read at any time. Cuttl notes that sequential tests need a little more traffic. Every test also shows its false-positive risk, and guardrail metrics and sample ratio mismatch checks run in the same report. AI assistants connected through the Cuttl MCP server can read the same results. ## Common questions ### What is sequential testing in simple terms? Sequential testing is a way of checking an A/B test while it runs without raising the false-positive rate. The test uses stricter rules early on, so you can stop sooner when the evidence is strong. ### Is sequential testing better than a fixed-sample A/B test? It is better when you check results often or need to stop harmful variants quickly. A fixed-sample test is more efficient when you commit to one look at the planned sample size. ### Does sequential testing let you stop a test early? Yes. The test can stop when a variant crosses the efficacy boundary, or for futility when an effect is unlikely. Early stops need a larger gap than a late one. ### What is the difference between sequential testing and peeking? Peeking means reading a normal test repeatedly, which inflates false positives. Sequential testing builds the repeated looks into the maths, so the error rate holds. ### What is mSPRT? The mixture sequential probability ratio test is an always-valid method. It gives a p-value that stays valid at every moment, without a planned number of looks. ### What is a holdout group, and why keep one after you ship a test? ### What is the flicker effect in A/B testing, and how do you stop it? ### How long to run an A/B test, and how to work it out