Sample Ratio Mismatch (SRM) Checker
Paste the visitor counts your test actually delivered. If the split is broken, the result is not close to right, it is void, and this tells you in one step which of the usual causes to go and look at first.
What your test delivered
Equal covers most tests. Custom is for a ramped rollout, a 90/10 holdback, or anything uneven.
The count each arm actually received. Paste straight from your dashboard, commas and all.
The verdict
The split is consistent with your plan.
Nothing here suggests the randomisation is broken. That is one failure mode ruled out, not a clean bill of health: this checks the split and nothing else.
- Chi-square
- 0.00
- Degrees of freedom
- 1
- p-value
- 0.958
- Total visitors
- 128,399
| Variants | Got | Expected | Off by | Share |
|---|---|---|---|---|
| Control | 64,209 | 64,200 | +10 | 50.0% |
| Variant B | 64,190 | 64,200 | -10 | 50.0% |
SRM check: 50.0/50.0 intended, 50.0/50.0 observed across 128,399 visitors. Chi-square 0.00 on 1 df, p 0.958: no SRM detected. Checked with the Optyv SRM checker.
If you have SRM, start here
Ranked by how often each one turns out to be the culprit, and grouped by the stage of the experiment it happens in. Work down the list: the confirm step is the specific thing to go and look at, not a general instruction to investigate.
The variant lives on a second URL, and everyone who abandons during the extra hop is counted as assigned but never counted as arrived. This is the most common cause by a distance, and the most reliably fixable.
How to confirm: Does the variant involve a redirect? Compare time to first byte between arms, and check whether the shortfall is concentrated on slow connections.
How SRM voids a test ↗The filter keys off one arm URL pattern, or runs after the variation script rather than before assignment, so automated traffic is stripped from one side and not the other.
How to confirm: Pull the counts before and after bot filtering. If the split is clean upstream and broken downstream, the filter is the cause.
Slow variation rendering makes that arm bounce harder, and a share of those bounces land after bucketing and before the analytics event fires.
How to confirm: Measure render time for the variation against control. Check whether the missing visitors cluster on mobile or on slower devices.
Why tests fail ↗Visitors are bucketed on page load, an audience condition excludes some of them a moment later, and the exclusion is not symmetrical across arms.
How to confirm: Check whether the targeting rule evaluates before or after bucketing, and whether it can depend on anything the variation itself changes.
A cached page or a stale edge response serves one arm to an entire segment of traffic regardless of the split you configured.
How to confirm: Look at the split by cache status and by edge location. A mismatch concentrated in one region or one cache state points straight here.
Assignment is correct in the testing tool and wrong in the warehouse, because the visitor identifier changed somewhere between the two systems.
How to confirm: Compare the raw counts in the testing platform against the counts in your warehouse. If only one of them has SRM, the join is the problem.
Run the QA checklist ↗Variants ramped at different times, one arm was paused and resumed, a second overlapping experiment shared the same traffic, or somebody forced themselves into a variant for testing. The taxonomy calls these interference, and they are the hardest to spot afterwards because the configuration looks correct by the time you check it.
How to confirm: Read the experiment audit log for edits after launch, and list every other test that ran on the same pages over the same window.
None of these? SRM can take a specialist days to trace, and the experiment is unusable in the meantime.
Send us your setup ↗What is sample ratio mismatch?
Sample ratio mismatch is when the traffic split your test actually delivered differs from the split you configured by more than chance can explain. You asked for 50/50 and got 48.1/51.9 on a hundred thousand visitors. That is not close enough, it is evidence that assignment or measurement is broken somewhere, and it invalidates the comparison rather than merely widening it.
It matters more than most data-quality problems because of what it implies. If visitors were not allocated the way you intended, the two groups are no longer exchangeable, so any difference between them can be selection rather than the change you made. A test with SRM does not have a noisy answer. It has no answer.
The percentage on its own tells you nothing, which is where most arguments about this go wrong. The same 48.1 against 51.9 is fatal on a hundred thousand visitors and completely unremarkable on two thousand. What changes is not the ratio but how unlikely that ratio becomes by chance as the sample grows, so only the p-value settles it, which is what the checker above computes.
A useful way to hold it: the check asks how often a working randomiser would produce a split at least this lopsided. If the answer is one run in five hundred thousand, the explanation that your randomiser is fine has become much less plausible than the explanation that something is quietly dropping or misrouting visitors. That is why the alarm threshold sits far below the 0.05 people are used to. This test runs on every experiment rather than once, so a one-in-twenty false alarm rate would have you chasing healthy tests permanently.
It is also worth knowing what a clean result does not buy you. No mismatch means the split is consistent with your plan, and nothing more. A test can pass this check and still be ruined by stopping early, by a metric that does not measure what you think, or by a promotion halfway through that changed who was visiting. Checking the split first is worth doing because it is fast and because the failure it catches is fatal rather than merely inconvenient.
How the check worksA chi-square goodness-of-fit test against your intended split, read at the threshold the experimentation literature settled on.
Each variant gets an expected count, which is the total multiplied by its intended share. The chi-square statistic adds up the squared gap between observed and expected, divided by expected, across every arm. Degrees of freedom is one fewer than the number of variants. The p-value is the chance of seeing a gap at least this large if the split were working correctly.
- Below 0.001, the split is treated as broken. That threshold is far stricter than the usual 0.05 because this check runs on every experiment you ever launch, so a 5% false-alarm rate would have analysts chasing ghosts constantly.
- Between 0.001 and 0.01, the result is not conclusive by convention but is worth investigating before you trust anything downstream.
- Above 0.01, nothing here suggests a problem with the split.
- If any variant expects fewer than five visitors, the chi-square approximation stops being trustworthy and the tool says so rather than quietly reporting a number.
One implementation note, because it changes what you see. Very broken splits produce p-values far below what a floating-point number can express as a difference from one, so this tool computes the tail probability directly rather than subtracting a cumulative probability from 1. That is why a badly broken 90/10 test reports a number here instead of a flat zero.
- Fabijan, Gupchup, Gupta, Omhover, Qin, Vermeer and Dmitriev, "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners", KDD 2019 (doi.org/10.1145/3292500.3330722). The source of the chi-square check and of the five-stage taxonomy the causes below are grouped by. Worth knowing: the paper itself sets no p-value threshold.
- Kohavi, Tang and Xu, "Trustworthy Online Controlled Experiments" (Cambridge University Press, 2020), chapter 21. The source of the stricter-than-usual threshold this tool alarms at, and of the argument for treating the split as a guardrail on every experiment rather than an occasional audit.
Common questions
It is when the traffic split your A/B test actually delivered differs from the split you configured by more than chance can explain. A 50/50 test that sends 48.1% of a hundred thousand visitors to control has SRM. It matters because it means the two groups were not assigned the way you intended, so any difference between them could be selection rather than your change, which makes the result unusable rather than merely imprecise.
Below 0.001 is the working convention, and it is deliberately far stricter than the usual 0.05. The reason is that this check runs on every experiment rather than once, so a 5% false-alarm rate would have you investigating perfectly healthy tests constantly. Kohavi, Tang and Xu set out the argument in chapter 21 of Trustworthy Online Controlled Experiments. Between 0.001 and 0.01 is worth a look without being conclusive.
Not by adjusting the analysis. The data is biased in a way you cannot measure from inside it, so reweighting or excluding the affected segment usually makes things worse by hiding the problem. The fix is to find the cause, correct it, and rerun the experiment. That is genuinely expensive, which is why checking the split before you look at the results is worth the ten seconds it takes.
Redirect-based variants losing visitors during the extra hop is the most common cause by a distance. After that: bot filtering applied unevenly across arms, flicker costing the slower variant real people, assignment firing before an eligibility rule, caching pinning one arm for a whole segment, and analytics joins dropping sessions between the testing tool and the warehouse. Overlapping experiments and mid-flight edits account for most of the rest.
It depends entirely on the sample size, which is exactly why the percentage alone never settles it. A 48.1 against 51.9 split is fatal on a hundred thousand visitors and completely unremarkable on two thousand. The same absolute gap becomes less and less likely by chance as the test grows, so a split that looked fine early can cross the line later without anything having changed.
No. It rules out one failure mode. A test can have a perfect split and still be broken by peeking at the results daily, by a badly defined metric, by running across a promotion that changed who was visiting, or by not having enough traffic to detect the effect it was powered for. The split is the first thing to check because it is cheap and fatal, not because it is the only thing.
Put this checker on your own site
Free to embed, no attribution deal and no tracking script. Useful next to any post about experiment QA, and it keeps a link back here.
<iframe src="https://optyv.com/tools/srm-checker/embed" width="100%" height="760" style="border:0;max-width:760px" title="Sample ratio mismatch checker" loading="lazy"></iframe>
<p>SRM checker by <a href="https://optyv.com/tools/srm-checker">Optyv</a></p>The iframe is a fixed 760px tall in this version. On a narrow column the checker scrolls inside the frame rather than clipping.
Publishing a result?
If you are reporting a test in public, this says the split was checked rather than assumed.
<a href="https://optyv.com/tools/srm-checker"><img src="https://optyv.com/badge/srm-checked.svg" alt="SRM checked with Optyv" width="200" height="40"></a>Found one you cannot trace?
SRM is the failure mode that costs the most time to diagnose and the most credibility to ignore. We build and QA tests so it does not happen, and we untangle it when it already has.
Talk to us ↗