The variation is up 5.2% on revenue per visitor, mobile largest contentful paint has moved from 2.4 to 2.9 seconds, and the checkout error rate has gone from 0.4% to 1.1%. That is not a judgement call. It is a decision with exactly four possible answers, and the only thing that decides which one you take is the cause of the breach rather than the size of the lift.
Which metrics to carry and how to wire them up is answered in the guardrail metrics post. This one is about the harder half: setting a threshold you can still defend on the morning a lift exists, reading a breach honestly enough to know whether it is real, and running the guardrail set across a whole programme rather than one test at a time.
What an ecommerce programme should guard
Page speed, returns and support contacts are the familiar three, and two more belong on the list that almost nobody carries. The first is error rate, meaning client-side exceptions and failed payment attempts on the tested template. Nearly every client-side variation adds script, script fails on the devices that were already struggling, and the failure lands as an abandoned checkout rather than as a lost conversion anybody can attribute. Measure it in percentage points, not in percent, because a move from 0.4% to 1.1% is a near tripling and reads like a rounding error.
The second is gross margin per order. A variation that lifts revenue per visitor by making a discount easier to find has produced a real result and a different business, and a readout that stops at revenue will never say so. Margin is also the guardrail most likely to be unavailable in the testing tool, which is not a reason to drop it: a monthly export of orders in the assignment list against cost of goods is enough, and it is the number a finance director will ask for eventually anyway.
How to set a guardrail threshold you can defend
A guardrail threshold is a degradation band, not a significance test, and the clearest statement of that comes from Microsoft’s analysis and experimentation team. In "A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments" (Dmitriev, Gupta, Kim and Vaz, KDD 2017), guardrails are defined as metrics "which are not clearly indicative of success of the feature being tested, but which we do not want to significantly harm when making a ship decision", and the working rule is stated just as plainly: "Small degradations in guardrail metrics can be expected", while "Large degradations, however, are generally not allowed". The paper is at exp-platform.com.
That reframing does real work. It means the question at threshold-setting time is not "is this change statistically significant" but "at what point does this stop being the ordinary cost of adding a feature and start being a decision somebody has to make". For speed, anchor the number to something external rather than to your own baseline: Google’s thresholds are 2.5 seconds for largest contentful paint, 200 milliseconds for interaction to next paint and 0.1 for cumulative layout shift, each read at the 75th percentile, and they are published at web.dev/articles/vitals. A variation that pushes a passing template into a failing band is a different conversation from one that makes a failing template slightly worse.
Why a guardrail that is not significant is not safe
Here is the misreading that costs the most, and it is almost universal. A test is sized so the primary metric can detect the effect you care about. The guardrails inherit that sample and nothing else, and several of them measure rarer events on smaller denominators, so they arrive with far less power than the metric the test was designed around. Returns is the worst case: a thirty-day cohort is a fraction of the traffic, arriving a month late.
The check takes a minute and belongs on every readout: compare the guardrail’s minimum detectable effect against its own threshold. If the minimum detectable effect is larger, the guardrail was incapable of catching the thing it was set to catch, and reporting it green is worse than reporting nothing, because green gets quoted. Say "not detectable at this sample" instead, and either pool the cohort over a longer window or accept that the guardrail is a monitoring commitment rather than a gate. The same arithmetic that sizes a test in the first place is what settles it.
The error runs both ways, which is why the list stays short. Kohavi and Longbotham recommend computing hundreds of metrics per experiment for the insights they give, while insisting in the same breath that the decision be made on a single overall evaluation criterion and that the false positive rate be controlled for. That is the shape to copy. Once enough numbers are being watched at 95% confidence some of them go red for no reason at all, so a guardrail set of five that each carry a written threshold produces arguments you can settle, and a dashboard of forty produces a weekly hunt for the one that moved.
What to do when a winner trips a guardrail
Four dispositions, and the cause of the breach picks one. Write which of them applies before the meeting, because the same four options discussed in front of a +5.2% chart become one option and three formalities:
- Ship anyway. Legitimate only when the guardrail was logged as a note rather than a veto and the movement sits inside the band you wrote down. If you find yourself arguing for it on a metric you set to stop, you are renegotiating rather than deciding.
- Fix the cause and rerun. The default when the breach comes from the change itself rather than from how it was built, because the fix alters what the variation is and the old result no longer describes it.
- Fix and ship without rerunning. Reserved for mechanical causes with no bearing on the behaviour under test: an eagerly loaded image, an unbatched request, a script that could have been deferred. State the reasoning in the archive so the exception stays an exception.
- Kill it. When the breach is the mechanism. A variation that lifts conversion by overstating availability is not fixable, because the overstatement is the feature.
Two organisational rules make those four survive contact with a good quarter. The decision is made by whoever owns the guardrail rather than whoever owns the roadmap, and it is recorded next to the result, so that a year later somebody can see which winners shipped with a caveat attached. Microsoft goes further and auto-shuts experiments that significantly degrade the product, which its own patterns of trustworthy experimentation describes alongside alerting the team automatically. Most ecommerce programmes are too small to automate a shutdown and easily large enough to automate the alert.
The guardrails only a whole programme can see
Every disposition above is a per-test decision, and the damage that ends careers is cumulative. Twenty winners shipped over a year, each adding 40 milliseconds and each comfortably inside a 10% ceiling, is 800 milliseconds nobody ever attributed to anything. No individual readout was wrong. The programme was, because the guardrail was only ever asked a question about one test at a time.
The fix is a standing measurement rather than a cleverer threshold. Re-baseline the guarded metrics quarterly from field data rather than from the last test’s control, and compare the current baseline against the one from four quarters ago; that comparison is the only place creep is visible. Then treat a drifted baseline as a piece of work with an owner rather than as a fact about the world. It is the same reason a testing programme keeps an archive at all.
Guardrails that are really data-quality checks
One category gets filed with guardrails and does not belong there. A guardrail vetoes a result; a data quality failure voids one. Sample ratio mismatch is the obvious member: a broken split does not mean the variation harmed something, it means every number in the report is describing a population that was never randomised, including the lift. Our SRM checker runs that test on your own counts in about a minute.
Keeping the two apart matters because putting them in one table invites somebody to weigh a validity failure against a lift, and that is not a trade that exists. Run the data quality checks first, refuse to read anything until they pass, and only then open the guardrail table. If you want the whole set built into your experiments by people who treat a red guardrail as a result rather than an obstacle, that is what our experiment engineering work is, and you can get in touch.
