Readouts that settle a question instead of reopening it

A significance verdict answers one narrow question about whether an observed gap is surprising. It puts no number on what the gap is worth, and worth is the only thing a business can act on. That number lives in the measured effect plus the range around it, which the NIST handbook defines as a span of values likely to contain the true one. Two results can clear the same threshold and mean entirely different things once you see how wide their ranges are.

That distinction is worth learning rather than delegating, and our article on what statistical significance actually claims works it through on real numbers. If your platform reports probabilities instead of p-values, the Bayesian and frequentist comparison explains what changes and what does not.

Calibration matters as much as method. The Microsoft and LinkedIn authors of Seven Rules of Thumb report that at a site running thousands of experiments a year most fail, and those that succeed move key metrics by 0.1% to 1.0% once diluted to overall impact. Against that, a claimed double-digit jump earns the scepticism the same paper attaches to Twyman’s law: any figure that looks interesting or different is usually wrong.

One artefact removes most of the arguments a readout can start. Before a variation goes live, four things are written down and circulated: the primary metric, the smallest effect worth acting on, the date the test stops, and the guardrails able to overrule a positive result. Agreeing all four while the answer is still unknown is what makes them binding afterwards.

The stop date is skipped most often, and it is a calculation rather than a preference. Our sample size and duration calculator turns your baseline and smallest useful effect into visitors and weeks; the runtime arithmetic behind it covers what to do when that answer is longer than anyone hoped.

Fixing the date is also what makes the significance threshold mean what the tool claims. Dmitriev and colleagues at Microsoft put it plainly in A Dirty Dozen: monitoring continuously and halting the instant a threshold is crossed raises the Type-I error rate, and they name both early stops and quiet extensions as common mistakes. They also warn that a significant reading on an underpowered metric tends to exaggerate the real effect, so a large win on a thin slice is a lead, not a result.

A guardrail is not a success metric. The same Microsoft team define guardrails as metrics that do not indicate whether the feature worked, but that you refuse to significantly harm in exchange for a ship decision, small movements expected and large ones not permitted. For a store that means returns and refunds, support contact rate, delivery promise accuracy, and paint timings on the pages the test touches.

Their thresholds belong in the pre-launch rule, because a limit set afterwards is a negotiation. Our guide to guardrail metrics names the five an ecommerce test should carry, and the four dispositions once one is tripped covers the awkward case where the winner is also the culprit. A guardrail that has never stopped anything is decoration.

The test of a readout is whether it holds up in front of somebody who sat in none of the meetings and has no reason to be generous. That reader asks three things in order: what did you say you would measure, what did you find, and what are you doing about it. A document that cannot answer all three is a slide deck.

So it opens with the disposition and the reasoning, reproduces the pre-launch rule unedited, then gives the effect with its interval, the guardrail panel against its agreed limits, the checks proving the arms were comparable, and any segment declared in advance. Our significance article publishes the reporting standard behind that shape, and it stays short enough that a finance director reads all of it.

One check sits ahead of the rest, because it can invalidate them. Where the split a tool actually delivered has drifted off the ratio you asked for, you do not have a result yet. Run that yourself with our free SRM checker before anyone reads a conversion number.

Winners are the minority of any honest programme. That is not a programme working badly, it is a programme finding out, and the value survives only if the finding is written where the next person looks. So every result leaves with a disposition and a row in the archive, dull ones included.

The archive is what stops a settled question being reopened by somebody who joined last month. Ours is free and ungated as the experiment archive template: twelve fields, a fixed tag vocabulary, worked examples. Take it and run this yourself. Our catalogue of the ways tests fail is mostly ideas the whole room felt certain about.

Teams buy this on its own, having built a programme they cannot get decisions out of. No tool changes and nothing is handed across. We will also say when the analysis is fine and the problem is upstream.

If the events feeding the numbers cannot be trusted in the first place, tracking and QA validation is the pass that comes before this one. If the shortage is builds rather than conclusions, A/B test development is the service that ships them, with the readout included in each.