A/B testing, also called split testing, is a method for proving that a change caused a result. One group of visitors sees the page you already have. A second group sees the page you are thinking of shipping. Whichever version moves the number you named in advance is the version that ships.
The reason it is worth the trouble is subtraction. Your conversion rate rose 9% last month. So did your paid spend, the weather, and the number of people who had already heard of you. A before-and-after comparison cannot separate those. A test can, because both groups live through the same month.
We have built 1640+ tests for DTC and mid-market brands across Europe and the USA, which is most of what our A/B test development work involves. This is what we wish every team knew before running their first one.
How a split test actually works
The mechanics are less impressive than the vocabulary around them. A testing tool intercepts each arriving visitor and assigns them, at random, to either the control or the variation. That assignment is sticky: if someone comes back tomorrow they see the same version they saw today, or the data is worthless.
Randomisation is the part doing the real work. It makes the two groups comparable without anyone having to think about it. Both end up with roughly the same mix of first-time and returning visitors, the same phone-to-desktop ratio, the same proportion arriving from paid search on a Tuesday afternoon. You have not controlled for those variables. You have made them cancel out.
The tool then counts one primary metric per group, plus a few supporting ones, and waits. Once enough people have been through, statistics tell you whether the gap between the two numbers is a real difference or the sort of gap you would get from flipping a fair coin a few thousand times. That is the entire apparatus.
Why it is worth the trouble
Decisions stop belonging to whoever is most senior in the room. Without evidence, the tie-breaker on any design argument is job title, and job title is not correlated with knowing what your customers will do. The rule we hold ourselves to is blunt: if a redesign cannot beat the control, it does not ship. It has killed work we were proud of.
Revenue becomes attributable. A well-run test does not tell you the new page is nicer. It tells you the new page is worth 6% more revenue per visitor, and if that page carries £2m a year, it tells you the change is worth £120,000. That is a number you can take to a board without flinching.
Big changes get de-risked. Replatforms and redesigns fail more often than anyone admits publicly. Testing lets you find that out on half your traffic for three weeks rather than on all of it for a year. The worst case of a losing test is a small, bounded, temporary dip. The worst case of an untested redesign is a slow erosion nobody ever diagnoses, because there is nothing left to compare it against.
And the learning compounds. Every test teaches you something about your customers, and the losers usually teach you more than the winners, because a loser kills an assumption the rest of the roadmap was quietly built on. A programme that has run for two years is not valuable for the winners it banked. It is valuable for the ideas it stopped you spending a quarter on.
The five kinds of test
Almost everything you run will be the first of these. The others exist for specific situations, and reaching for them when the situation does not call for it is a reliable way to spend a quarter of your traffic learning nothing.
- Classic A/B test: control against one variation. The cleanest read available, and the right default for most changes.
- A/B/n test: control against two or more variations. Useful when you genuinely cannot choose between ideas, and expensive, because the traffic requirement climbs with every arm you add.
- Multivariate test: combinations of several element changes at once, to find which combination performs best. Needs far more traffic than most stores have, and belongs on very high-volume pages only.
- Split URL test: two different addresses rather than two versions of one page. The right tool for full redesigns, new landing pages and template changes.
- Server-side test: the difference is rendered by your server rather than by the visitor browser. Necessary for pricing, search ranking, cart logic and anything else buried in the stack.
Client-side against server-side is the distinction that actually matters technically. Client-side tests rewrite the page in the browser with JavaScript: quick to build, ideal for visual changes, and prone to flicker, where the visitor glimpses the original for a fraction of a second before it changes. Server-side tests avoid flicker entirely and can reach logic JavaScript cannot touch. Mature programmes run both.
The process we run on every test
Seven steps, in this order. The order is not decoration. Most of the failures we get asked to diagnose are one of these steps done late, or skipped entirely.
- Research. Mine analytics for drop-off, watch session recordings, read support tickets and reviews. You are looking for friction, not for ideas.
- Hypothesis. Write down what you observed, what you will change, and what you expect that to do to which metric.
- Prioritise. Score the backlog on impact, confidence and effort, and start where the traffic already is: product pages, cart, checkout.
- Build. Clean code, cross-browser QA, and every goal tracked correctly before anything goes near a visitor.
- Calculate and launch. Work out the numbers you need first, then run for at least one full business cycle.
- Analyse. Significance first, then segments, then the metrics you were not hoping to move.
- Ship, document, iterate. Winners go into the codebase. Losers go into the archive with whatever they taught you.
A hypothesis has three parts: because we observed this, we believe that this change, for these visitors, will result in this movement in this metric. Fill in the first clause with real evidence or do not run the test. "Because 68% of mobile visitors leave the product page without ever reaching the reviews" is a hypothesis. "Because the page feels cluttered" is a preference wearing a lab coat.
Step four is where programmes quietly die. A variation that flickers, breaks in Safari or adds 400ms to the page is not testing your hypothesis, it is testing your bug, and it will usually lose. Most of the tests we are handed to rescue were never wrong about the idea. They were wrong about the build.
Step five is the one teams skip, because it is arithmetic rather than opinion. Work out the sample size you need before you launch, not after the numbers start looking interesting, and fix the end date at the same time. Then run to it. One full business cycle is the floor, usually two to four weeks, because your customers behave differently on payday than they do on a wet Tuesday.
What this looks like on a real store
Patterns repeat across stores often enough to be worth knowing, and never often enough to be worth trusting. Moving review stars directly under the product title on mobile wins frequently on stores with real review volume. Free-delivery threshold messaging tends to move average order value more than it moves conversion rate, and where you place it matters more than how it is worded. Payment icons and a returns line near the checkout button punch above their weight on colder traffic.
Then there is the one that keeps us in work: the beautiful new product page template losing to the ugly old one. We see it constantly. Simplified mobile filters win on large catalogues and lose on small ones, which is the whole argument in miniature. A best practice is a hypothesis with good PR. Which of these apply to your store is a separate question, and one we have written up at length for ecommerce A/B testing.
The statistics, minus the maths degree
Start with statistical significance, the probability that the gap you measured is not noise. The convention is 95%, which does not mean a 95% chance the variation is better. It means that if your change did nothing whatsoever, you would see a result this extreme less than one time in twenty. Calling tests at 80% because the line looked promising is the fastest route to a programme built on winners that were never there.
Sample size is how many visitors you need before that calculation means anything at all. Low-traffic stores are not shut out of testing, but they are shut out of testing small things: the smaller the change, the more people you need to see its effect through the noise.
Minimum detectable effect is the same idea from the other end. Given your traffic and your timeframe, there is a smallest lift you could reliably notice. If that floor sits at 10% and you are hoping to find 2%, the test has already failed and nobody has launched it yet.
Peeking is the quiet killer. Checking daily and stopping the moment significance appears does not speed the test up, it inflates your false-positive rate, because you have given yourself twenty chances to catch a random fluctuation instead of one. Decide the duration up front. Look when it ends.
Tools matter less than teams think
The platforms worth knowing are Optimizely, VWO, AB Tasty, Convert, Kameleoon and Dynamic Yield, plus Shopify-native options such as Shoplift, and Intelligems if you are testing price. Google Optimize was retired in 2023, and most brands who relied on it have landed on one of these; our hands-on tool comparison goes through all seven.
The honest position is that the choice barely matters. Every serious platform can split traffic and compute a p-value. Programmes succeed or fail on research quality, hypothesis quality, build quality and velocity, none of which are features you can buy. We have seen strong programmes running on cheap tools and useless ones running on expensive tools, and never once the reverse for the reason people expect.
The mistakes that invalidate a programme
None of these announce themselves. Each produces a result that looks exactly like a real one, which is what makes them expensive, and most of them happen before the test goes live.
- Testing without a hypothesis. Random changes produce random learnings, and nothing you can build on next quarter.
- Calling it early. See peeking, above. The dashboard will tempt you every single day it is open.
- Testing tiny changes on small traffic. You will wait four months for a result you still cannot read.
- Ignoring flicker and build bugs. A broken variation does not test an idea, it tests a bug.
- Watching conversion rate alone. A variation can lift conversion and lower average order value at the same time; revenue per visitor is the metric that catches it.
- Running one test and concluding that testing does not work. Industry-wide, roughly one test in four or five wins. It is a volume game.
- Leaving winners in the testing tool. Implement them in the codebase, or they add JavaScript weight to every page and eventually break.
Does A/B testing hurt SEO?
No, provided you do it the way Google has explicitly said to do it. The question comes up in almost every kickoff, usually from someone who has been told otherwise by a tool vendor.
Show Googlebot the same thing you show people: cloaking is the actual violation, and it is a different activity from testing. On split URL tests, put a canonical tag on the variation pointing back to the original, so the two addresses are not competing with each other. Use temporary redirects rather than permanent ones for those tests, because the arrangement genuinely is temporary.
And end your tests. An experiment that has been running for eleven months is not an experiment, and of everything in this section it is the only one that causes real problems in practice.
In-house, or with a partner
Testing in-house works when you have a developer who can build variations properly, an analyst who can read them, and enough traffic to keep both busy. Those three rarely sit free in a DTC team of fifteen. What happens instead is that tests get built slowly by whoever has capacity, QA’d optimistically, and analysed by the person who proposed them. The full decision framework is in our outsourcing guide.
That last one is the real argument for an outside team. We have no ego invested in whether your redesign wins, which makes it considerably easier for us to tell you that it lost. The other two arguments are velocity, because tests running in parallel compound faster, and build quality, because flicker-free and correctly tracked is a craft rather than a checkbox.
If your programme has stalled somewhere between the third test and the first honest readout, get in touch and we will look at it with you.
