The 20-point A/B test QA checklist
The list a variation has to pass before it is allowed near real traffic, in the order we run it, with nothing removed to make it look tidier than it is. It is for whoever signs a test off, and it exists to stop a build shipping with a broken goal that will report a clean, confident result about the wrong thing.
The checklist
0 of 20 ticked
0 of 20 ticked
Before you touch the code
Five checks that cost minutes now and weeks later. Every one of them exists because a test was once launched without it and had to be thrown away.
Rendering and devices
The variation is now built. This group is what separates a build that survives contact with real traffic from one that looks perfect in a desktop emulator.
Tracking, which is where the real damage lives
A broken variation looks broken and gets fixed within the hour. A broken goal produces a clean, confident report about the wrong thing, and can survive all the way to a shipping decision. It is the largest defect category and the least visible one.
Traffic, assignment and the first hour
QA does not end at launch. The first sixty minutes of live traffic tell you more than any staging environment can, and the checks are quick.
One rule makes the twenty work: whoever wrote the variation does not sign it off.
Ticks are for this sitting only and are not saved anywhere. A checklist that remembers looks authoritative about a build it knows nothing about, so if you need a record against the test, take the Word file below and keep it with the ticket.
Take it with you
The Word file is the same twenty checks with a box beside each one, ready to be edited, renamed for your own stack and kept alongside the ticket. No address needed for it.
Download Word (.docx)Generated from this page, so the two cannot disagree.
If you would rather have it printed and pinned up, the PDF is the one thing here we ask for an address for.
How we use it
The reviewer runs the same list every time and records the result against the test, so a strange number three weeks later can be traced to what was and was not verified. Self-review reliably misses the defects that follow from an assumption the author still holds, which is exactly the class of defect that survives to production, which is why the sign-off goes to someone else.
The list is fixed on purpose. A checklist that gets shortened for easy tests stops being a checklist and becomes a judgement call about which tests deserve care, and that judgement is always made by the person least able to make it: the one who just finished the build and wants it live. It grows in one direction only. When a defect reaches production, the check that would have caught it is added permanently, and the list is a little longer for everyone afterwards.
Of the defects it catches, tracking and goal configuration accounts for roughly 31%, ahead of responsive breakage at 24%, flicker at 18%, app and theme script conflicts at 14%, audience and targeting errors at 8% and everything else at 5%. Tracking leads, and tracking is the group most often skipped, because a broken layout announces itself and a broken goal does not.
Expect about ninety minutes for a straightforward visual variation and half a day for anything touching cart, checkout or pricing. If that sounds expensive, price it against a month of traffic spent on a test that measured a tracking bug. The reasoning behind each group, and what to do when a check fails, is in the full article on the QA checklist we run on every build.
Version
- v1.0 · 2026-08-07First published. Twenty checks in four groups, as run.
Questions about the checklist
That is what the Word file is for. The twenty here are the ones that survived contact with a lot of ecommerce builds, but the specific in-app browsers you care about, the devices your traffic actually uses and the analytics platform you reconcile against are yours. What we would keep is the shape: four groups in that order, and no shortening for easy tests.
Because a checklist with a pass mark invites a team to launch on eighteen out of twenty, and the two skipped are always the slow ones, which are tracking and real-device rendering. Every check is a gate rather than a point. If one cannot be met, that is a decision to record against the test, not a score to average away.
No, deliberately. The state lives in the page and is gone when you leave it. A checklist that remembers looks like a record of a build it knows nothing about, and a half-remembered one is worse than none. If you need a record, download the Word file and keep it with the ticket, where it can be attached to the test it belongs to.
About ninety minutes for a straightforward visual variation, and half a day for anything touching cart, checkout or pricing. If QA on a real test is taking twenty minutes, it is not QA, it is a glance at a desktop browser. The cost to weigh it against is not the ninety minutes, it is a month of traffic spent measuring a tracking bug.
Using this on your own site
The template is free to quote, screenshot and link to. If it is useful, a credit link is the thing that keeps it findable for the next person looking for a checklist rather than a lead magnet.
<p>A/B test QA checklist by <a href="https://optyv.com/templates/ab-test-qa-checklist">Optyv</a></p>No tracking in either snippet, and no nofollow on the link. It is a credit, not a beacon.
Or use the badge
For a build log, a readme or a process page, to show which checklist a test was run against.
<a href="https://optyv.com/templates/ab-test-qa-checklist"><img src="https://optyv.com/badge/qa-checklist-used.svg" alt="QA checked with the Optyv A/B test checklist" width="240" height="40"></a>Want the checklist run for you?
We build and QA variations for ecommerce teams who would rather not own the boring half. Same list, every test, signed off by someone who did not write the code.
See how we build tests ↗