A test that has not been QA’d is not a test. It is a bug with a dashboard, and it will still report a winner with a confidence figure next to it. This is the actual pre-launch checklist we run before a variation is allowed near real traffic, in the order we run it, with nothing removed to make it look tidier than it is.
It is deliberately boring, and its boredom is what makes the interesting arguments about hypotheses and statistics worth having at all. How a team resources this sort of work, who writes the variation code and what the market charges for it, belongs to our guide to CRO development. This post is only the list, and the list is the part most teams are missing.
Before you touch the code
Five checks that cost minutes now and weeks later. Every one of them exists because a test was once launched without it and had to be thrown away:
- The hypothesis names one primary metric, and that metric is configured as the primary in the tool before any code is written.
- The sample size and expected duration are calculated and written down, so nobody discovers at week three that the test could never have reached a verdict.
- The control is the current live page, not a staging copy, and any pending theme or app release that would change it is either paused or scheduled around.
- Every element the variation touches is identified by a stable hook rather than a nth-child position, so a merchant reordering blocks does not silently break the test.
- The kill criteria are agreed in writing: what result, guardrail breach or defect stops the test early, decided while everyone is still calm.
Rendering and devices
The variation is now built. This group is what separates a build that survives contact with real traffic from one that looks perfect in a desktop emulator:
- The variation renders correctly on real hardware, not simulated viewports: at minimum one mid-range Android device and one older iPhone, on a throttled connection.
- Safari, Chrome and Firefox are all checked, plus in-app browsers from the traffic sources that matter, because Instagram and email clients render differently and carry real revenue.
- There is no visible flicker on a cold cache and a slow connection, measured by recording the load rather than by watching it on office broadband.
- The page still works with the variation partially applied, so that a slow script produces a plain page rather than a broken one.
- Console errors are zero on both arms, and any third-party app the variation shares the page with has been loaded alongside it rather than assumed harmless.
Tracking, which is where the real damage lives
A broken variation looks broken and gets fixed within the hour. A broken goal produces a clean, confident report about the wrong thing, and can survive all the way to a shipping decision. Of the defects this checklist catches, tracking and goal configuration accounts for roughly 31%, ahead of responsive breakage at 24%, flicker at 18%, app and theme script conflicts at 14%, audience and targeting errors at 8% and everything else at 5%. It is the largest category and the least visible one:
- The primary goal fires exactly once per conversion, verified by completing a real transaction on each arm and reading the raw payload rather than a dashboard summary.
- Revenue is passed with the correct value, currency and tax treatment, and it reconciles against the back office for the same test transactions.
- Every secondary and guardrail metric is configured before launch, because a metric added mid-test only has data from the moment it was added.
- The experiment identifier reaches your analytics platform, so results can be reproduced outside the testing tool that has an interest in the answer.
- Consent and privacy tooling does not block the assignment or the goal for a subset of visitors, which is a common and quietly asymmetric source of missing data.
Traffic, assignment and the first hour
QA does not end at launch. The first sixty minutes of live traffic tell you more than any staging environment can, and the checks are quick:
- The split matches the configuration, checked as a ratio rather than eyeballed, and rechecked the following morning when overnight bot traffic has had its say.
- Assignment is sticky: the same visitor sees the same arm across pages, sessions and devices where your identity resolution claims it should.
- Traffic is arriving on both arms from every source you expect, since a source missing entirely from one arm is a targeting bug rather than a slow start.
- Conversions are appearing on both arms within the first hours at roughly the volume the baseline predicts, which catches a broken goal before a full day is wasted.
- Guardrail metrics have a threshold and an owner, so a revenue or page-speed regression stops the test automatically rather than waiting for the weekly readout.
The sign-off
Twenty checks in four groups, and one rule that makes them work: whoever wrote the variation does not sign it off. Self-review reliably misses the defects that follow from an assumption the author still holds, which is exactly the class of defect that survives to production. The reviewer runs the same list every time and records the result against the test, so a strange number three weeks later can be traced to what was and was not verified.
The list is fixed on purpose. A checklist that gets shortened for easy tests stops being a checklist and becomes a judgement call about which tests deserve care, and the judgement is always made by the person least able to make it, which is the one who just finished the build and wants it live. It grows only in one direction: when a defect reaches production, the check that would have caught it is added, permanently, and the list is a little longer for everyone afterwards. That is the whole mechanism by which a testing team gets better at the boring half of the work.
Expect about ninety minutes for a straightforward visual variation and half a day for anything touching cart, checkout or pricing. If that sounds expensive, price it against a month of traffic spent on a test that measured a tracking bug. Two of these checks reward a post of their own, and eliminating flicker properly is the one teams most often declare done without measuring. If you want this discipline running on your builds without hiring for it, experiment engineering is the service that carries it, and we are easy to reach.
