A variation that lifts add to cart by 12% while adding 800ms to your product page and pushing 30-day returns from 11.2% to 14.8% is not a winner. It is a loan, taken against a quarter that has not happened yet, at an interest rate nobody wrote down. Guardrail metrics are how you refuse the loan, and they only work if somebody agreed them before the result existed.
A guardrail metric is a number your test is not allowed to damage while it moves the number it was built to move. It does not choose the winner. That job belongs to the primary metric, which on an ecommerce test is almost always revenue per visitor. A guardrail has exactly one power: a veto. The discipline is picking five of them and writing down, in advance, what happens when one goes red.
Guardrail, secondary, primary: three different jobs
These three get used interchangeably in readouts and they are not interchangeable. The primary metric decides. A secondary metric explains, so when checkout completion rises you read payment method mix to understand why, and nothing in that reading carries a veto. A guardrail constrains, and it is picked for a specific reason: it names the way this particular variation could cheat. A countdown timer could lift conversion by borrowing from returns. A personalised recommendation block could lift average order value by borrowing 400ms from every visitor, including the ones who never scroll far enough to see it. Choose each guardrail by asking where this change would take the money from if it were dishonest.
That framing is also what stops the list growing. A guardrail set of fifteen metrics is a dashboard, and dashboards get scrolled past. Five is roughly what a team will genuinely read on a Friday, and five is enough, because the ways an ecommerce test can borrow money are not that varied.
The ecommerce five
These are the five we carry on almost every build, with the thresholds we start from. Treat the numbers as defaults to argue with rather than constants: a fashion store already running a 40% return rate needs a tighter returns ceiling than a store selling kettles.
- Mobile largest contentful paint, measured on the tested template rather than site-wide. Ceiling: the variant may not run more than 10% slower than the control. Speed is the guardrail most often skipped and most often breached, because nearly every client-side variation adds weight and nobody ever attributes next quarter’s decline to it.
- Bounce rate on the tested template. Ceiling: 2 percentage points above control. This is the cheapest early warning of a build defect, because flicker, layout shift and a broken mobile breakpoint all surface here days before they surface anywhere else.
- Average order value. Floor: no more than 5% below control. Anything that makes it easier to buy one cheap item can lift conversion while shrinking the basket, and a conversion-only readout calls that a win every single time.
- Return or refund rate at 30 days, restricted to orders placed by visitors who were actually in the test. Ceiling: 2 percentage points above control. This is the guardrail that catches persuasion doing damage: sizing claims, urgency, and photography that flatters the product past the point of accuracy.
- Support contacts per thousand orders, tagged by topic wherever your helpdesk allows it. Ceiling: 15% above control. A variation that removes a delivery estimate to tidy up a layout does not lose conversion; it moves the question into a ticket, and the cost lands on a team who were never in the readout.
Thresholds get set before launch, or they get negotiated after
Every guardrail needs three things written down before traffic starts: the threshold, the owner and the action. The threshold is a number, not a direction. The owner is a named person rather than a team, because a guardrail owned by everyone gets overruled by whoever is most enthusiastic about the result. The action is one of stop, hold or note: stop kills the test automatically, hold pauses the rollout pending a conversation, note logs the movement and lets the test run. All three belong in the same document as the hypothesis and on the pre-launch checklist, so that a missing guardrail is a build defect rather than an oversight.
The reason for front-loading all of it is political rather than statistical. A threshold agreed in advance is arithmetic. The same threshold proposed on the morning a variation is showing +12% is a negotiation, and the side holding the lift wins that negotiation roughly always, usually with a sentence about how the returns number is probably seasonal.
Wiring them up without starting a data project
None of the five needs new infrastructure, which is where teams go wrong by assuming a warehouse has to come first. Speed comes from your real-user monitoring, or a synthetic run against both variants keyed by the test cookie. Bounce and average order value come out of the testing tool you already pay for, provided order value is passed as revenue rather than as a plain conversion event. Returns come from your returns app or the order admin, exported once a month against the assignment list. Support contacts come from a helpdesk tag applied by hand for the duration of the test, which costs an agent about four seconds a ticket and is worth every one of them.
The awkward one is lag. Returns arrive 30 days behind the orders that caused them, so a test finishing on the 14th has no returns reading until the following month. The answer is not to wait, and it is certainly not to drop the guardrail. Ship the winner but keep the assignment list, then read the returns cohort when it matures and treat a breach as a rollback trigger rather than an interesting fact. A guardrail with a lag is still a guardrail. It just fires late.
A save, in figures
A homeware client tested a complete the room block on product pages: three complementary items placed below the buy button. Add to cart rose 12% and conversion rose 4% at 96% confidence, which is a good result by any honest reading of the primary metric. Two guardrails went red. Mobile largest contentful paint moved from 2.3s to 3.1s, an 800ms penalty against a 10% ceiling, because the block loaded its own imagery eagerly. Then the returns cohort matured, and 30-day returns on orders containing a bundle came back at 14.8% against a store baseline of 11.2%, because the complementary items were being bought off a thumbnail and a price with no dimensions anywhere near them.
The rebuild took a day: imagery deferred below the fold, the block rendered from product data already on the page rather than a second request, and dimensions added to every item card. The rerun landed add to cart at +9%, largest contentful paint at 2.4s, and 30-day returns at 11.8%. That version shipped. The first version, on a conversion-only readout, would also have shipped, and it would have cost more in returns handling than the extra orders were worth while making every product page slower for everybody.
Guardrails are not caution, and a team who reads them as caution will drop them the first time one blocks a favourite idea. They are what makes an aggressive programme survivable: you can try genuinely risky things once you have decided in advance what would make you stop. Some surfaces need their own set on top of the five, and email popups are the obvious case. If you want the five wired into your builds by people who treat a red guardrail as a result rather than an obstacle, that is what our experiment engineering work is, and you can get in touch.
