Experiment Planner
Most A/B tests fail before they launch — underpowered, called early, or run so long they drift with the season. This planner sizes the test properly from your baseline rate and the smallest lift you care about, then turns sample into a real timeline and the discipline to read it honestly.
A sound experiment is sized before it starts. From your baseline conversion rate and your minimum detectable effect (the smallest relative lift worth catching), a two-proportion z-test gives the required sample per variation; daily traffic turns that into a duration. Run it in whole weeks, fix the sample size up front, and do not peek — stopping at the first significant blip inflates false positives. Pre-register guardrail metrics so a win does not hide a loss.
Experiment Planner inputs and result
How to use this calculator
- Enter your baseline rateUse the control’s current conversion rate for the metric you are testing. This anchors the whole calculation — lower baselines need dramatically more sample.
- Set the minimum detectable effectChoose the smallest relative lift worth detecting. Be honest: a 2% MDE sounds rigorous but can require an impractically huge test. A bigger MDE means a faster, feasible one.
- Add daily traffic and armsEnter total daily users entering the test and how many variations (including control). The planner converts required sample into a real duration in days and weeks.
- Choose confidence and power95% confidence and 80% power are the standard defaults. Raising either increases the sample and the time needed; lower them only with eyes open.
- Read the plan and follow the rulesYou get sample per variation, total sample, and duration, plus the two rules that decide validity: run whole weeks with no peeking, and pre-register guardrail metrics.
RGM Expert Says
The hard truth about A/B testing is that the result is mostly decided before launch. We watch teams run a test for ‘a couple of weeks’, see green, and ship — with no idea the test was powered to detect only a 40% lift that never realistically existed. Sizing the experiment first, from a baseline and an honest MDE, is what separates a real experiment from a coin flip with a dashboard.
The number people fight over is the MDE, and they should. A small MDE feels rigorous but can demand a sample your traffic cannot deliver this quarter. We reframe it as a business question: what is the smallest lift that would actually change a decision? Often that lift is larger than people assume, and once it is set honestly the test becomes feasible. We would rather power a test to catch a 10% lift in three weeks than chase a 2% lift for a year.
Then there are the two rules we never bend: no peeking, and guardrails. Peeking — checking daily and stopping at the first significant reading — quietly turns a 5% false-positive rate into something far worse, because you get many chances to cross the line by luck. And every test gets a small set of guardrail metrics, so a checkout-conversion win that secretly tanks average order value or refunds gets caught before it ships.
How it works
Required sample per variation comes from the two-proportion z-test: the planner computes the alternative rate from your baseline and MDE, combines the critical values for confidence (zα/2) and power (zβ), and solves for the sample needed to tell the two rates apart. Total sample multiplies by the number of arms; duration divides total sample by daily traffic.
- p₁ — baseline conversion rate (control).
- MDE — minimum detectable effect, the smallest relative lift to catch.
- zα/2 — critical value for confidence (1.96 at 95%).
- zβ — critical value for power (0.84 at 80%).
- Daily traffic — total users/day entering the test, split across arms for duration.
The two-proportion z-test sample formula and the no-peeking / guardrail discipline follow standard experimentation practice, notably Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments. This tool sizes and schedules the test; it does not analyze results.
Why most A/B tests fail before launch
An A/B test answers one question: is the difference between variants real, or noise? It can only answer that if it has enough sample to separate signal from chance. Skip the sizing step and you get an underpowered test — one that quietly lacked the data to detect the effect you were hoping for, so a real winner reads as ‘no difference’ and gets killed. Powering the test up front is the difference between a decision and a guess.
The biggest lever is the minimum detectable effect, and it is a business decision dressed as a statistical one. Halving the MDE roughly quadruples the required sample, because you are asking the test to resolve a finer difference. The right MDE is the smallest lift that would actually change what you do — usually larger than teams assume, which is exactly what makes a test feasible.
Two practices protect the result once it is running. No peeking: fix the sample size in advance and do not stop at the first significant reading, because checking repeatedly gives random noise many chances to cross the line and inflates false positives. And guardrails: pre-register a primary metric plus a few safety metrics, so a headline win cannot quietly damage revenue, refunds, or speed without anyone noticing.
How inputs move the test
A feel for the trade-offs before you commit. These directions hold for the two-proportion z-test.
| Change | Effect on sample | Practical note |
|---|---|---|
| Halve the MDE | ~4× larger | Smaller effects are expensive to prove |
| Lower baseline rate | Larger | Rare events need more traffic |
| 95% → 99% confidence | Larger | Buy certainty with sample |
| 80% → 90% power | Larger | Fewer missed real effects |
| Add a variation | More total + longer | Each arm needs full sample |
What practitioners say about experiment design
Decide the sample size before you launch and do not stop early. Peeking at the data and calling the test on the first green reading is how you ship false positives.
The minimum detectable effect is a business question. Power the test for the smallest lift that would actually change your decision, not the smallest you can imagine.