Testing vendors turned a statistics preference into a holy war, largely because it is one of the few things left to differentiate a pricing page. The uncomfortable part is that on the same dataset the two frameworks almost always point at the same decision. Where they genuinely differ is in what each one lets you say out loud, and in how each behaves when you get impatient, which is the one failure mode neither of them can save you from.
What follows assumes you already know what a p-value is claiming. If you do not, our guide to statistical significance in A/B testing does that job properly and this will make more sense afterwards. This post is about the choice between frameworks, and the conclusion is unfashionable: it matters far less than whether your team can leave a running test alone for three weeks.
What each one is actually claiming
A frequentist p-value is a statement about data, not about your variation. It says: if the two versions were truly identical, how often would random assignment throw up a gap at least this large? Below 5%, the result gets called significant. Nothing in that sentence is a claim about the probability that the variation is better, which is why almost every non-statistician who quotes a p-value quotes it wrong, usually in a way that flatters the test.
Bayesian analysis answers the question people meant to ask. Given this data, and given a starting assumption about which uplifts are plausible, what is the probability that the variation beats the control, and how much do I stand to lose if I ship it and I am wrong? That starting assumption is the prior, and it is where the objections come from. In commercial tools the prior is weak and uninformative, and the data overwhelms it within a few thousand visitors, so it is both less scandalous and less magical than the argument suggests.
One test, read twice
Take an ordinary shape of result: 20,000 visitors per arm, the control converting at 2.4% and the variation at 2.7%. That is a relative lift of 12.5%, and it is exactly the sort of number that starts an argument on a Thursday afternoon.
The frequentist reading gives a two-sided p-value of 0.057. Just outside the line, so the formal answer is that the test has not reached significance and should either keep running or be logged as inconclusive. The Bayesian reading of the identical numbers gives roughly a 97% probability that the variation is better than the control, plus an expected loss of about 0.002 percentage points of conversion rate if you ship it and it turns out you were wrong.
Those two readouts look like a disagreement and are not. They are the same evidence expressed on two scales, and the reason one clears its bar while the other misses is that the bars were set by different conventions rather than by different data. The practical difference is what a stakeholder hears. Nobody in the history of ecommerce has been talked out of a decision by ‘p equals 0.057’. ‘A 97% chance of being better, with almost nothing to lose if we are wrong’ gets a decision made in the room, which is a genuine advantage and a genuine hazard in the same sentence.
What actually changes in practice
Five things change when you switch frameworks, and accuracy is not among them:
- Language. Bayesian output is directly quotable: probability to beat control, expected loss, a credible interval that means what a reader assumes it means. Frequentist output needs a translator, and the translation is usually wrong.
- Stopping. A Bayesian tool will happily let you stop once expected loss drops below a threshold, which sounds like a licence to watch the dashboard and is not one.
- Small effects. Expected loss makes it easy to ship a tiny win on the grounds that being wrong costs nothing. Defensible for a one-line change, reckless for anything that adds code you will maintain for two years.
- Priors. Almost nobody sets one deliberately, so the honest description of most Bayesian ecommerce testing is frequentist arithmetic wearing friendlier labels.
- Losses. Frequentist reporting treats an inconclusive test as a non-event. Bayesian reporting hands you a 61% probability to beat control and tempts you to call it a lean, and a lean is not a result, it is a coin with a chip in the paint.
Peeking is the sin neither forgives
Both frameworks assume you fixed the rules before you saw the data. Break that assumption and both break with it. Checking a fixed-horizon frequentist test five times across its run takes the false positive rate from the 5% you designed for to somewhere near 14%, and checking it daily for a fortnight pushes it past 25%. A naive Bayesian tool with a flat prior inflates in the same direction for the same reason: many chances at a threshold, and you keep the first one that clears it.
The real fix is not a framework, it is a stopping rule fixed in advance, or an engine built for continuous monitoring: sequential testing, always-valid confidence intervals, or a Bayesian implementation with a decision threshold you commit to before launch. Those exist in both traditions and both work. What exists in neither is a way to watch a live dashboard, stop on the good day, and keep the answer.
Which engine your tool hands you
You mostly do not choose. Most platforms ship one engine and offer no switch. VWO’s SmartStats is Bayesian, Optimizely’s Stats Engine is a sequential frequentist design built specifically to make continuous monitoring valid, GA4’s experiment reporting leans Bayesian, and a large share of the Shopify-native apps in our roundup of ecommerce testing tools do not document their method anywhere, which tells you more about how much statistical thought went into them than any engine name would.
So pick the tool on assignment quality, on how cleanly it joins to your analytics, and on whether it can run server-side when a test needs that. Those three decide whether your data is real. The engine layered on top decides only how the same evidence gets phrased in the readout.
Discipline beats framework, and it is not close
Across the 1640+ tests in our archive we can count on one hand the calls where the framework itself changed the decision. What changed decisions repeatedly was duller: a sample size agreed before launch, one primary metric named in the brief, a review date in the calendar, and the willingness to write ‘inconclusive’ in the log rather than squint at segments until a winner appears.
Use whatever your tool ships, learn to state its output honestly, and spend the argument budget on the run rules instead. If the framework debate is the liveliest thing in your programme, the programme has a different problem from the one the debate describes, and the fix is experiment engineering discipline rather than a migration. We are easy to reach if you want a second opinion on how your last three tests were called.
