Experimentation has a jargon problem: half the terms sound interchangeable and the other half sound scarier than they are. The result is meetings where four people use the word significant and mean three different things, and a losing test gets shipped because nobody wanted to ask what SRM stood for.
These are the forty terms that actually come up, each defined in plain English. If you are new to the discipline entirely, read what A/B testing is first; this page is the reference you keep open afterwards.
A note on what got cut. The experimentation literature carries hundreds of terms, most of which exist to make conference talks sound harder than the work is. The filter for this list was simple: does the word come up in real client conversations, in tool interfaces, or in the moment a result is being argued about? Anything that failed all three stayed out, however impressive it sounds.
A to C
- A/B test. Two versions of a page shown to randomly split traffic at the same time, judged on one metric agreed in advance. The foundation everything else on this page builds on.
- A/B/n test. The same idea with more than one variation against the control. Each extra arm raises the traffic you need, which is why most of these should have been two separate tests.
- AOV, average order value. Revenue divided by orders. Watch it alongside conversion rate: a variation can lift one while quietly lowering the other.
- Bayesian and frequentist. Two statistical frameworks for reading a test. Frequentist asks how surprising the data is if nothing changed; Bayesian updates a probability that the variation is better. Tools pick one for you; discipline matters more than the choice.
- Bounce rate. The share of visitors who leave without a second interaction. A diagnostic metric, and a poor primary metric for a test, because you can lower it without selling anything.
- Click-through rate. Clicks divided by views of the thing clicked. Useful for a component, dangerous as a verdict: more clicks into a checkout that does not convert is not a win.
- Client-side testing. The variation is built in the visitor browser with JavaScript after the page arrives. Quick to build, ideal for visual changes, and the source of flicker when done badly.
- Confidence interval. The range the true effect plausibly sits in. A lift of 4% with an interval of 1% to 7% is a real result honestly stated; the interval is the honesty.
- Control. The current experience, left alone. Half the value of a test is that the control forces your existing page to defend itself.
- Conversion rate. Orders, signups or goals divided by visitors. The most quoted number in ecommerce and rarely the right primary metric on its own.
- CRO, conversion rate optimization. The practice of systematically raising the share of visitors who buy. The strategy half is research and prioritisation; the engineering half has its own guide.
- CUPED. A variance-reduction technique that uses pre-experiment data to make tests read faster. Worth knowing exists; worth implementing only once the basics are boringly reliable.
D to L
- False positive and false negative. Calling a winner that is not real, and missing a winner that is. Every design choice in testing trades one against the other.
- Feature flag. A switch that turns code on for some users and off for others. The plumbing behind most server-side testing.
- Flicker, also FOUC. The visitor glimpses the original page before the variation renders. The test then measures the delay rather than the idea, and the variation usually loses for the wrong reason.
- Guardrail metric. A metric a test must not damage while it moves the primary one: page speed, refund rate, support contacts. Winners that breach a guardrail are not winners.
- Holdback. A slice of traffic kept on the old experience after a rollout, to check the win survives contact with reality. The difference between shipped and proven.
- Hypothesis. Because we observed this, we believe this change for these visitors will move this metric. If the first clause is missing, it is a guess with paperwork.
- ICE and PIE scoring. Prioritisation frameworks scoring ideas on impact, confidence and ease, or potential, importance and ease. The point is not the acronym; it is that the backlog has an order someone can defend.
- Lift. The relative change the variation produced: 2.6% to 3.0% is a 15% lift, not 0.4%. Always state which arithmetic you are using.
M to R
- MDE, minimum detectable effect. The smallest lift your traffic can reliably notice in a sensible time. If your MDE is 10% and you expect 3%, the test is unreadable before it launches.
- Multivariate test. Combinations of several element changes tested at once to find the best mix. Needs far more traffic than most stores have; earns its place only on very high-volume pages.
- Novelty effect. Returning visitors click the new thing because it is new, inflating early results. One more reason tests run in full weeks, not exciting first days.
- P-value. The probability of data at least this extreme if the change did nothing. Not the probability you are right, however much everyone wants it to be.
- Peeking. Checking results daily and stopping the moment significance appears. It multiplies false positives, because you gave chance twenty chances instead of one.
- Personalization. Different experiences for different audiences. Powerful after the fundamentals are validated, and a multiplier of unvalidated guesses before that.
- Power. The probability a test detects an effect that genuinely exists. Underpowered tests miss real winners and breed the belief that testing does not work.
- Primary metric. The one number, agreed before launch, that decides the test. Chosen after the fact, it is not a metric, it is a justification.
- Randomization. Visitors assigned to control or variation by chance, and kept there. It is what makes the two groups comparable without anyone controlling for anything.
- Revenue per visitor. Revenue divided by all visitors, converting or not. The ecommerce north star, because it catches a conversion lift that cannibalises order value.
S
- Sample ratio mismatch, SRM. The split between arms drifts from the split you configured: 48.7 against 51.3 when you asked for 50/50. It means the assignment is broken and the result, however pretty, is invalid.
- Sample size. The number of visitors a test needs before its verdict means anything, calculated before launch from baseline, MDE and significance. Doing this maths last is doing it never.
- Segmentation. Reading a result by device, traffic source or customer type after the overall verdict. Where the interesting findings live, and where false positives breed if you go fishing.
- Server-side testing. The variation is rendered by your server before the page is sent. No flicker, deeper reach, more engineering. Pricing, search and checkout logic live here.
- Split URL test. Control and variation live at different addresses and traffic is routed between them. The right tool for full redesigns and template changes.
- Statistical significance. The convention, usually 95%, that a result this size would appear by chance less than one time in twenty. A floor for believing a result, not a certificate of one.
T to W
- Type I and Type II error. The formal names for false positive and false negative. If a vendor says their platform eliminates both, close the tab.
- Test archive. The written record of every experiment, losers included, with what it taught you. The most valuable asset a programme produces and the first thing lost when an agency changes.
- Variation. The changed experience being tested against the control. One clear idea per variation; a variation carrying six changes produces a verdict about nothing.
- Winner implementation. Rebuilding a winning variation as real code in the theme or codebase, out of the testing tool. Skipped, it becomes permanent JavaScript weight that eventually breaks.
Three of these forty do most of the damage in practice. Peeking and sample size have full guides of their own: how significance actually works and how to calculate the sample a test needs. SRM is subtler and worth a word here, because it is the one that looks most like good news. A test with mismatched arms often shows a beautiful lift, produced entirely by the bug doing the mismatching: a redirect that fails on old browsers, a bot filter applied to one arm, a variation that crashes for a subset of devices. The uplift is real in the dashboard and imaginary in the bank account. Check the split before you check the result, every time.
Using the vocabulary without being used by it
Jargon has two jobs: precision among people who share it, and intimidation of people who do not. In sales conversations it mostly does the second job. A vendor who says our Bayesian engine reaches significance faster is making a claim you can now unpack: faster than what, at what false-positive rate, and does faster matter when the bottleneck in most programmes is build capacity rather than maths?
The same defence works internally. When a stakeholder announces that a test was significant, ask which metric, at what interval, over what duration. When a report says the variation won on mobile, ask whether that segment was named before launch or discovered afterwards. Neither question requires statistics; both require only the definitions above, used at the moment they are being blurred. Most bad testing decisions are not maths errors. They are vocabulary errors that nobody in the room felt confident enough to catch.
The vocabulary that matters is the working ten in the FAQ below, plus the discipline to use them in order: hypothesis before build, sample size before launch, significance before verdict, archive before the next test. That order is most of the engineering side of CRO, and it is teachable in an afternoon. If a term you keep meeting is not on this page, it is either niche or marketing; get in touch and we will tell you which.
