Multivariate testing: how it works and when to use it

What multivariate testing is

Multivariate testing (MVT) changes several parts of one page at the same time and tests every combination of them, to find which mix works best and which part matters most. Test two photos, two headlines and two button labels on a product page and you get 2 × 2 × 2 = 8 versions, each shown to an equal share of visitors.

An A/B test compares whole versions of a page. A multivariate test splits the page into sections, gives each section a few options, and lets the combinations run against each other. The original page, with every section unchanged, is the control.

How the number of combinations grows

The number of versions in a multivariate test is the number of options in each section multiplied together. That product grows fast, which is the main thing to plan around:

• 2 sections with 2 options each: 4 combinations.
• 3 sections with 2 options each: 8 combinations.
• 2 sections with 3 options and 1 with 2: 18 combinations.
• 4 sections with 3 options each: 81 combinations.

Testing every combination is called a full factorial design. Some tools offer a partial or fractional design that runs only a planned subset and estimates the rest. It needs less traffic, but it can miss how two changes affect each other, which is often the reason for running a multivariate test in the first place.

How much traffic a multivariate test needs

A multivariate test needs about as much traffic as an A/B test to tell you which option wins in each section, and many times more to pick a winning combination. The difference comes from how the visitors are pooled.

Take a product page that converts at 5%, where you want to detect a 10% relative lift (5.0% to 5.5%) at 95% confidence and 80% power. A standard sample size calculation gives about 31,000 visitors per group.

• An A/B test of one change: 2 groups, so about 62,000 visitors in total.
• A multivariate test of 3 sections with 2 options, read section by section: every option is seen by half the visitors across 4 of the 8 combinations, so the same 62,000 or so answer “which headline is better?” for all three sections at once.
• The same test, read combination by combination: each of the 8 combinations needs its own 31,000 visitors, so about 250,000 in total.

So the question you want answered sets the traffic you need. If 250,000 visitors would take months on your page, test fewer sections, or run the sections as separate A/B tests. The guide to statistical significance explains confidence and power in more depth.

How to read multivariate test results

Multivariate results answer two questions: which option in each section does best on average, and whether one combination beats the rest. Read them in that order.

• Section by section. For each section, compare its options across every combination that contains them. A headline that lifts the conversion rate whichever photo and button sit next to it is a finding you can rely on.
• Combination by combination. Compare each combination with the control. This is where interactions show up, such as a benefit headline that only helps when the lifestyle photo is above it.
• Interactions. If the best combination isn’t simply the best option from each section put together, two sections are affecting each other. Confirm that pattern with a follow-up A/B test before you rely on it.

With many combinations, a few will look good by chance. Treat a combination that wins by a small margin with care, and watch metrics such as bounce rate and revenue per visitor as guardrails.

Multivariate test, A/B test or split URL test?

Use a multivariate test when you have several independent changes on one page and enough traffic to learn which ones matter. Use an A/B test for most other questions. Use a split URL test when the new version is a different page.

• A/B test: one change, or a few versions of a whole page. Fastest to reach a result, and the right start for most sites. See what A/B testing is.
• Multivariate test: two to four sections of one busy page, such as the photo, headline and call to action above the fold on a product or landing page.
• Split URL test: a redesign or a new template, served from its own URL.

Pages with low traffic are rarely a good fit, because every extra option divides the visitors further. Pricing, checkout and high-traffic landing pages usually are.

Multivariate tests in Cuttl

In Cuttl’s Web Experiments, a multivariate test is one of the test types, next to A/B, multi-page and split URL tests. You set it up in sections, and Cuttl turns every combination into a version.

• Each section has options, and the first option is the original. In the visual editor you edit only the part of the page that section is about.
• Every combination is tested and gets an equal share of the traffic, up to 32 combinations per test. The combination where every section keeps its original is the control.
• The report shows each combination against the control, and a section analysis: how each option performs across every combination that contains it, with its conversion rate and uplift over the section’s original.
• As with other tests, you can watch recordings of visitors in each version to see why one did better.

Multivariate tests are on the Max plan. AI assistants connected through the Cuttl MCP server can draft a multivariate test from a list of sections and options for a person to review.

Common questions