What sequential testing is
Sequential testing is a way of analysing an A/B test while data is still coming in, with statistical rules that keep the false-positive rate at the level you chose, usually 5%. A fixed-sample test is only valid if you read it once, at the planned sample size. A sequential test stays valid however often you look, so you can stop as soon as the evidence is strong enough.
Why checking a normal A/B test early goes wrong
A standard significance test assumes one look at the data. Every extra look gives chance another opportunity to cross the 5% line, so the real false-positive rate climbs. In a simulation published by an engineering team at a large streaming company, testing just twice at a 5% level pushed the effective rate to about 10%. Checking a dashboard every day for three weeks does far more damage.
The result is a familiar pattern. A test shows “95% significant” on day four, the team ships the variant, and the lift never shows up in the numbers. Early gaps are mostly noise, and stopping on the first good-looking one selects for noise. Our guide to statistical significance explains what the 95% figure does and does not promise.
How sequential testing keeps the error rate under control
Sequential methods spend the allowed error gradually. Instead of one 5% threshold at the end, the test uses a stricter bar early on and relaxes it as evidence builds up, so the total chance of a false positive across all looks stays at 5%.
- Decision boundaries. The test sets an efficacy boundary (stop, the variant is better) and often a futility boundary (stop, there is unlikely to be an effect).
- Error spending. The 5% is shared out across the looks, so an early look can only stop the test with very strong evidence.
- A running statistic. Each new batch of visitors updates the statistic, which is compared with the boundary again.
An early stop needs a large gap. A late stop needs only a modest one. That asymmetry is what lets a clearly better or clearly worse variant end the test in days while a marginal one runs to the end.
Two families: group sequential and always valid
Most tools use one of two designs, and they suit different data:
- Group sequential tests plan a fixed number of looks, such as once a day for four weeks, and set a boundary for each. They have the most statistical power when you can estimate the maximum sample size in advance, and they suit daily batch reporting.
- Always-valid tests, such as the mixture sequential probability ratio test (mSPRT), produce a p-value or a confidence sequence that stays valid at every moment. You do not need a planned number of looks or an exact sample size, which makes them a better fit for streaming data and for teams that check results whenever they like.
The price of that flexibility is power. In published comparisons, always-valid methods need somewhat more traffic than a group sequential design that has the right sample size, and both need more than a single fixed-sample test that is read exactly once at the planned end.
What sequential testing costs you
Sequential testing trades a little precision for the freedom to look. The trade-offs are worth knowing before you switch:
- Wider early intervals. A confidence sequence is wider than a normal confidence interval at the same sample size, so early estimates of the lift are loose.
- More traffic for small effects. If the real lift is small, a sequential test may need more visitors than a fixed test read once at the end.
- Overstated lifts on early stops. A test that stops because it crossed the boundary tends to report a lift that is larger than the true one. Treat the size of an early win as an upper estimate.
- Not a cure for bad setup. A sample ratio mismatch, a changing audience or a promotion that runs mid-test will mislead a sequential test just as they mislead any other.
When to use sequential testing
Sequential testing fits when the cost of waiting or of a bad variant is high and results are checked often:
- High-stakes changes, such as pricing or checkout, where a clearly harmful variant should stop quickly.
- Teams that look at results daily, who would otherwise be tempted to stop early by hand.
- Tests on pages with enough traffic that a strong effect shows up within days.
A fixed-horizon test is still a good choice for small optimisations on a page with steady traffic, where the plan is to wait for the planned sample and read the result once. Bayesian methods are a third option, and they handle repeated looks differently, as the guide to what A/B testing is outlines.
Sequential testing in Cuttl
In Cuttl’s Web Experiments, the statistics step of a test offers three models: Bayesian (the default), Sequential and fixed-horizon frequentist. Sequential uses always-valid p-values based on a mixture sequential probability ratio test.
- The smallest effect worth detecting, the minimum detectable effect (MDE), is set before launch, and the test uses it to tune the statistic.
- Cuttl checks daily and keeps the lowest always-valid p-value seen so far, so the result never goes back to “not significant” after a lucky day.
- Lift ranges come from a confidence sequence, which stays valid at every look. When several variants are compared with the control, an optional Bonferroni correction splits the error across them.
- The results page shows progress against the planned sample and says when a sequential test is safe to read at any time. Cuttl notes that sequential tests need a little more traffic.
Every test also shows its false-positive risk, and guardrail metrics and sample ratio mismatch checks run in the same report. AI assistants connected through the Cuttl MCP server can read the same results.
