# What is a holdout group, and why keep one after you ship a test? > A holdout group is a share of visitors kept on the original after a change ships. Learn how it measures the real effect, how big to make it and when to end it. Source: https://cuttl.io/blog/holdout-group/ · Last updated: 2026-11-04 A holdout group is a share of visitors kept on the original after a change ships. Learn how it measures the real effect, how big to make it and when to end it. ## What a holdout group is A holdout group is a share of visitors deliberately kept on the original experience while everyone else gets a change. The two groups run side by side over the same weeks, so the gap between them is the change’s real effect. For example, a shop that rolls a new shipping note out to 90% of visitors keeps the other 10% on the old page and compares add-to-cart rates. ## How a holdout group differs from a control group A holdout group and a control group are built the same way: visitors are picked at random and see the original. The difference is timing. A control group exists while a test is running, to decide whether a variant wins. A holdout group continues after the decision, once the winner has been rolled out to everyone else. The same word is used in marketing for people kept out of an email, ad or personalization campaign, so the team can see what the campaign added beyond what would have happened anyway. ## Why keep a holdout after a test has a winner A finished test is a forecast, and a holdout is how you check it. The result came from a few weeks of traffic. Once the winner is live, seasonality, a promotion, a new traffic source or the novelty of a fresh design can all move the numbers, and none of them tells you whether the change itself still helps. Comparing the weeks before and after a launch has the same weakness, because anything else that changed in those weeks gets mixed into the answer. That method is still useful when a site has too little traffic to split, as covered in A/B testing with low traffic. A holdout removes the problem: both groups live through the same weeks, so outside events hit them equally. It also gives a team something concrete to report. “Add to cart is higher than in the holdout” is easier to defend than “add to cart went up after we shipped.” Web Experiments. See how Cuttl handles holdout groups. ## How big should a holdout group be? There is no single right size. It is a trade between speed of the answer and the lift you give up on the held-back visitors. Many teams start with 5 to 10 percent of visitors. - A smaller holdout costs fewer conversions but takes longer to show a clear difference, because few visitors are in the comparison. - A larger holdout answers sooner but keeps more visitors on the version you believe is worse. - The effect you want to detect matters as much as the share. A small lift needs many visitors in both groups, which is the same maths as minimum detectable effect and how long to run an A/B test. ## How to run one without fooling yourself A holdout only works if it stays a fair comparison. Four habits keep it that way. - Assign visitors at random and keep each visitor in the same group in every session. A person who sees the old page on Monday and the new one on Friday is in neither group. - Use the same eligibility rules for both groups, so the same pages, devices and audiences are compared. - Read the result after both groups have enough visitors, not after the first good day. - Change nothing else for the holdout. If the held-back visitors also miss a promotion, the gap is no longer about your change. End the holdout when the question is answered, by moving the rollout to 100%. Holding people back from a winner forever costs conversions for no new information. ## Holdout groups in Cuttl In Web Experiments, a holdout belongs to a rollout. When a test has a winner, you roll that version out to a share of eligible visitors, and the rest stay on the original as the holdout. On the Max plan a holdout is on by default, at 10% of visitors, and you can change the share. On Pro a rollout goes to everyone, so there is nothing to compare it with, and the Free plan has no rollouts. Cuttl compares the new version with the holdout once both groups have 30 visitors, and says plainly when there is no significant difference yet. If you change the rollout’s share while it runs, the comparison is weighted so each group stands for the whole time it ran. To end the holdout, move the rollout to 100%. For the step before, see what A/B testing is. After a rollout, Journey Insights recordings of both groups show how visitors actually used each version. ## Common questions ### What is a holdout group? It is a share of visitors kept on the original experience while everyone else gets a change, so the two groups can be compared over the same period. ### What is the difference between a holdout group and a control group? A control group is the original version during a test. A holdout group is kept after the test, once the winning version has been rolled out to everyone else. ### How big should a holdout group be? There is no fixed size. A smaller share costs fewer conversions but needs more time to show a clear difference. Many teams start with 5 to 10 percent. ### Does a holdout group cost conversions? Yes, a little. The held-back visitors stay on the version you expect to be worse. That cost is the price of knowing the real effect, and you can end the holdout once the question is answered. ### Are holdout groups on the free plan in Cuttl? No. Rollout holdouts are on the Max plan. On Pro a rollout goes to all eligible visitors, and the Free plan has no rollouts. ### What is the flicker effect in A/B testing, and how do you stop it? ### How long to run an A/B test, and how to work it out ### Minimum detectable effect: what it is and how to set it