Every quarter we audit testing programmes that "did not work". The tooling is almost never the problem. Nine times out of ten the test was dead before a single visitor saw it, killed by one of three habits that feel completely harmless at the time.
1. The hypothesis was not one
"Let's try a green button" is a guess with no failure condition attached. A hypothesis names the friction, the fix and the number that should move, and it commits to all three before you look at anything.
Written properly it reads like this: because exit surveys show delivery anxiety on the product page, showing the delivery date above the fold will raise add-to-cart by at least 5%. If you cannot fill that sentence in, the test cannot teach you anything — you will ship the variant when it is up and shrug when it is down, which is what you would have done anyway.
The fix is a written backlog where every idea carries its evidence, ranked by expected impact. We reject roughly half of client test ideas at that stage. The rejection is free; discovering the same thing through six weeks of traffic is not.
2. The sample maths came last
Most underpowered tests were doomed at the whiteboard. A page converting at 2.4%, tested for a 5% relative lift, needs around 210,000 visitors per variant. Run that on a template seeing 30,000 a month and the "two-week test" is really a fourteen-month one.
What you get instead is an inconclusive result that gets filed as a loss. Do the power calculation before you brief the designer — the full method is here — and accept that sometimes the honest answer is that this page cannot be tested.
3. Somebody peeked
Day four. Variant B is up 12% and somebody senior wants to ship it. Stopping a test when it "looks significant" is the fastest way to manufacture winners out of noise, because with enough looks almost any test crosses the line at some point on its way to nowhere.
Pre-register the metric, the sample size and the end date, then leave it alone. We hide dashboards for the first week for exactly this reason. Why significance breaks when you peek is worth understanding properly, because the instinct to check is very hard to argue down without it.
The failure nobody counts: winning the wrong metric
There is a fourth failure that never gets recorded as one. The test wins on its chosen metric and the business does not move. Add-to-cart goes up 9% and revenue is flat, because the change pulled people into the cart who were never going to buy.
Pick a primary metric as close to money as your traffic allows. Micro-conversions are useful as diagnostics and dangerous as decisions. If you must optimise add-to-cart because revenue is too noisy, say so out loud in the readout so nobody mistakes the proxy for the goal.
The failure that happens after the test
There is a fifth failure, and it is organisational rather than statistical: the winning variant that never ships. It happens more than anybody admits, and it wastes the entire cost of the experiment.
The test ran in a testing tool, as an overlay on the live site. Winning means somebody now has to build it properly in the theme, get it reviewed, and deploy it — and that work sits in a different queue, owned by a different team, competing with the roadmap. Six weeks later the experiment is still running as a client-side patch, slowing the page it improved.
Agree before launch who implements a winner and roughly when. A test whose result nobody has committed to acting on is not an experiment, it is a survey — and surveys are much cheaper to run.
The same applies to losers. Turn them off promptly and record what was learned somewhere a future colleague will find it, because the second most expensive test is the one somebody re-runs in eighteen months having never heard about the first.
The pre-flight checklist
Five questions, all answered in writing before a test goes live. If any one is unanswerable, the test is not ready:
- What friction are we fixing, and what evidence says it exists?
- What is the primary metric, and what change in it would count as a win?
- How many visitors per variant do we need, and how long is that at current traffic?
- What is the end date, and who is allowed to stop it early?
- What will we do if it loses — and is that different from what we would do having never run it?
If you recognise more than one of these in your own programme, start with the hypothesis backlog rather than the tooling. It is the cheapest to fix, it exposes the other three problems on its own, and it can be done this week with a document and an argument.
Most of the fixes here happen before anything reaches a browser. They are written down in a document, argued about in a meeting, and settled with a number rather than a preference. That is unglamorous work, and it is the difference between a testing programme and a queue of redesigns with a stopwatch attached to each one.
It is also worth saying plainly that a programme running ten well-designed tests a year will beat one running forty badly-designed ones, every time. Volume is the metric agencies report because it is easy to count. Decisions changed is the metric that correlates with revenue, and it is almost always a much smaller number.
None of these failures are technical, which is why buying better testing software fixes none of them. They are failures of process, and the fix in each case is something written down before launch rather than argued about afterwards.
The programmes that compound share one unglamorous trait: they treat a test as a decision with a cost, not as a feature to be shipped. That framing kills roughly half the ideas before they consume any traffic, and the half that survive are worth the six weeks.
That last question is the one that saves the most money. If a loss changes nothing, the test was never a decision — it was reassurance. Our broader guide to ecommerce A/B testing covers the rest of the programme; if you would rather we looked at yours directly, get in touch.
