Multivariate testing is the feature everyone asks for in the kickoff call and almost nobody has the traffic to use. It is not a difficult idea and it is not a con. It is arithmetic, and the arithmetic is unkind: the thing that makes MVT attractive, testing several elements at once, is the same thing that multiplies the traffic bill until the schedule stops being a schedule and becomes a hope.
This post settles one decision and no others: multivariate test, or a sequence of A/B tests on the same page. If you want the tour of the test types themselves, what A/B testing is covers all of them properly. Here we are only doing the sums, because the sums are what the decision actually turns on.
What multivariate testing does
An A/B test compares two versions of a page. A full-factorial multivariate test builds every combination of several element variants and runs them simultaneously, so each element earns an individual read and, more importantly, so interactions between elements become visible. Interaction is the honest case for MVT: a new headline can win on its own, a new hero image can win on its own, and the two together can cancel because the page now says the same thing twice in a row.
Every combination is its own arm, and every arm needs its own sample. That one sentence is the entire cost structure, and everything below is a consequence of it. The confusion that sells MVT is the assumption that four elements means four times the work. It means two to the power of four, and exponents do not negotiate with quarterly planning.
The arithmetic, with real numbers
Four elements with two versions each: a headline, a hero image, a call-to-action label and a delivery badge. That is two to the fourth, sixteen combinations. The count gets underestimated constantly because the interface asks you for four small decisions rather than sixteen large ones.
Now the sample. At a 2.4% baseline conversion rate, detecting a 10% relative improvement at 95% confidence and 80% power needs roughly 64,000 visitors per arm. Sixteen arms is about 1,024,000 visitors to that single page. A store sending 50,000 monthly visitors through that template is looking at twenty months, during which the page must not change while your catalogue, your prices and two Christmases will.
That estimate is generous, too, because it ignores the multiple-comparison problem sitting inside every MVT. Sixteen arms means sixteen chances to clear a threshold. Tools that correct for it need more traffic still; tools that do not correct for it will hand you a winner that is noise wearing a rosette. Our note on A/B test sample size has the general version of this calculation, and it is worth running before anyone opens the MVT builder.
Fractional factorial buys speed with an assumption
The standard rescue is a fractional factorial design, usually sold under the Taguchi name: run a chosen subset of the sixteen combinations and infer the rest. It does genuinely cut the traffic requirement, sometimes by more than half, and the maths behind it is sound. It pays for the saving by assuming the interactions are small enough to ignore, which is awkward, because interaction detection is the only thing full-factorial MVT does that a sequence of A/B tests cannot. A design that assumes away its own reason to exist is a hard thing to recommend with a straight face.
What four A/B tests give you instead
Run the same four elements as four sequential A/B tests. Each has two arms at 64,000, so 128,000 visitors per test and 512,000 in total: half the traffic of the multivariate version, about ten months at the same 50,000 a month, and a completely different shape to those ten months.
You get a decision every two and a half months rather than one decision in twenty. Each winner ships immediately and starts earning, so the baseline the next test measures against is already improved and the compounding runs in your favour. Each loser dies early and cheaply. And the reasoning stays legible: when somebody asks in a year why the delivery badge is there, the answer is a test with a date on it rather than a cell in a factorial grid nobody can reconstruct.
The sequence is also more forgiving in ways the grid is not:
- A misconfigured arm ruins one test rather than sixteen, and you find out in weeks rather than at the end of a year.
- You can reorder the queue when the business changes, which across twenty months it certainly will.
- Seasonality is contained inside one short window instead of being smeared across a year and a half of trading.
- Every result lands in the archive as a clean one-variable statement that somebody can act on two years later without rerunning it.
- A stalled test can be stopped and its traffic handed to the next question, whereas abandoning a factorial design halfway through leaves you with sixteen underpowered arms and nothing to report.
The three cases where MVT genuinely wins
- Serious traffic on one template. Not a busy store overall: 500,000 or more monthly visitors to the specific page, so sixteen arms fill in weeks rather than seasons.
- A specific interaction hypothesis, written down. Not “let us see which combination works”, but a stated reason to expect two elements to fight each other or amplify each other.
- Elements that genuinely cannot be separated, where changing one forces another to change with it and testing them independently would test something you would never actually ship.
All three are hypothesis conditions rather than curiosity conditions, and that is the tell. Running a multivariate test to find out what happens is the most expensive form of curiosity experimentation offers. Fewer than 2% of the 1640+ tests in our archive are multivariate designs, and every one of them sat on a page with traffic to spare and a written reason to expect interaction. The rest of the requests that arrived asking for MVT were served better, and faster, by a queue.
What to tell the person who asked for it
The answer is not no, it is the number. Open a sample size calculator, put in your baseline, the smallest improvement worth having and your real traffic to that one template, multiply by the number of combinations, and read the months out loud. The conversation usually ends there, and it ends without anybody having to be the person who is against ambition.
If the answer comes back inside three months, run the multivariate test properly: a written interaction hypothesis, QA on all sixteen arms, and a correction for the multiple comparisons. If it does not, a sequence will beat it on every axis that matters. Getting either built cleanly is A/B test development work rather than a configuration exercise, so tell us what you are trying to learn and we will do the arithmetic with you before anyone builds anything.
