Why most changes aren’t A/B tested
A/B tests need enough visitors to give a clear answer, and some changes can’t be split anyway, such as a new menu across the whole site or an urgent fix. So most teams make the change and then look at the numbers.
Why a simple before-and-after can mislead you
Comparing “four weeks before” with “four weeks after” sounds fair, but plenty of other things change at the same time: seasons, holidays, ad campaigns, a newsletter going out. If sales rose across your whole site, your change may have had nothing to do with it.
A better way: compare against the rest of your site
Instead of looking only at the page you changed, look at how it moved compared with the pages you didn’t change. If both went up by the same amount, something else caused it. If the changed page went up and the rest stayed flat, your change is the likely reason. This is known as a difference-in-differences comparison.
- Use whole weeks on both sides, so every weekday counts equally.
- Skip the first day or two after the change while caches and visitors catch up.
- Wait until both sides have enough visits and conversions to compare.
Reading the answer
A careful check gives you one of four answers: improved, worse, inconclusive (the data can’t tell yet), or too early. “Inconclusive” is a real answer too. It tells you the change didn’t make a big difference either way.
When an A/B test is still better
If the page gets plenty of traffic and the decision matters, an A/B test is the stronger option, because it runs both versions side by side at the same time. Before-and-after checks are the next best thing when a test isn’t practical.
Before-and-after checks in Cuttl
Cuttl’s Change Impact notices when your pages change, with no release notes needed, and compares the changed pages with the rest of your site before and after. It says clearly when there isn’t enough data yet, and how many weeks an A/B test would need instead.
