A/B Test

An A/B test is a controlled experiment that compares two versions of something — a web page, an email subject line, an ad, a button, a price — by randomly dividing the audience between them and measuring which version performs better on a predefined metric. Version A is usually the current one, the control; version B is the variation, the challenger. Visitors are assigned at random to one or the other, each sees a single version, and the experiment records how each group behaves against the metric that matters, such as click-through, conversion rate, or sign-ups. Because the only systematic difference between the two groups is which version they saw, any reliable difference in their behavior can be attributed to the change. That is the whole logic: isolate one change, split the audience fairly, and let real behavior, not opinion, decide which version wins.

Random assignment is what makes the test trustworthy. By splitting traffic at random, an A/B test balances out all the other factors — device, time of day, traffic source, mood — across both groups, so they differ only in the variant they received. This is what turns a comparison into causal evidence rather than a guess. A/B testing replaces argument and intuition with measurement: instead of debating which headline is stronger, you run both and read the result. It is the workhorse of conversion optimization, product development, and marketing, precisely because it answers the question 'does this change actually help?' with data. Its power is also its limit — it tells you which of the versions you tested did better, not why, and not whether some untested third option would beat them both.

An A/B test changes one thing and compares two versions, which keeps the result clean and easy to interpret. A multivariate test is different: it varies several elements at once — say a headline, an image, and a button color together — and tests many combinations to find which mix performs best and how the elements interact. Multivariate testing can reveal interactions an A/B test cannot, but it needs far more traffic, because the audience is split across many more combinations, and it is harder to read. The trade-off is clarity versus richness. An A/B test gives a clean answer about one change quickly; a multivariate test explores many changes at once but demands much more traffic and care. Most teams start with A/B tests and reserve multivariate testing for high-traffic pages where the extra complexity pays off.

An A/B test is also distinct from a simple before-and-after comparison, and that distinction is the source of its rigor. Comparing this week's performance to last week's after making a change is not an A/B test, because the two periods differ in countless uncontrolled ways — season, traffic mix, news, day of week — so any difference could come from those rather than the change. An A/B test runs both versions at the same time on randomly split traffic, holding everything else constant, which is exactly what before-and-after cannot do. Statistical rigor matters here: a result is only trustworthy when the sample is large enough and the test runs long enough to reach statistical significance, so the observed difference is unlikely to be chance. Calling a winner early, on too little data, is the classic way an A/B test misleads rather than informs.

Running an A/B test well begins before any traffic is split. Form a clear hypothesis — what you are changing and why you expect it to help — and pick a single primary metric that defines success, so you are not fishing through results afterward for any number that moved. Change one thing at a time in a true A/B test, so a win can be attributed to a specific cause. Decide the sample size and duration in advance, run the test long enough to reach statistical significance, and resist the strong temptation to stop the moment the numbers look good, because early peeks at noisy data produce false winners. Run both versions simultaneously on randomly assigned traffic, and account for normal cycles like weekday-versus-weekend behavior by running across full periods rather than a convenient few days.

The failures are well known and easy to commit. Calling a winner too early, before the result is statistically significant, is the most common — random noise looks like a real difference until enough data accumulates. Testing too many changes at once in something billed as an A/B test makes it impossible to know which change caused the result. Ignoring statistical significance and acting on a difference that is within the margin of chance leads to confident decisions built on nothing. Running for too short a window misses weekly patterns and seasonal effects. And cherry-picking a metric that happened to move, rather than the one you set out to improve, turns a rigorous method into self-deception. The discipline is a clear hypothesis, one change, a single metric, a pre-set sample and duration, and patience to wait for significance — which is what separates an A/B test that informs a decision from one that merely flatters a hunch.

An A/B test, also called a split test, applies the logic of the randomized controlled experiment to marketing and product decisions, comparing two versions on randomly split traffic.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where a/b test is a core concern: