Test design & hypothesis
A test’s trustworthiness is decided before it launches. This module is the craft of design: writing a structured, falsifiable hypothesis; committing to one primary metric with guardrails; isolating variants; choosing the right randomization unit and audience; sizing duration from the sample math; and QAing the experiment like the deploy it is.
What you will learn
Why test design decides the result
A test’s trustworthiness is mostly decided before it launches, by its design. A clear hypothesis, one primary metric with guardrails, the right randomization unit, a properly-sized audience, and a planned duration are what separate a result you can act on from a number that fools you. The most common reason A/B tests mislead isn’t bad statistics at the end — it’s sloppy design at the start that no analysis can rescue.
Think of design as the part you fully control. You can’t control whether your idea wins, but you can control whether the test gives a trustworthy answer either way. Get the design right and a loss is as useful as a win; get it wrong — ambiguous metric, contaminated randomization, under-powered audience — and even a ‘winner’ is just noise wearing a confidence interval.
Claim: Trustworthy experiments depend on disciplined design and validity checks; getting an answer is easy, getting one you can trust is hard. Source: Kohavi, Tang & Xu — Trustworthy Online Controlled Experiments. Context: The leverage is up front: hypothesis, metric, randomization, sample size, and QA — not in post-hoc analysis.
A weak hypothesis is ‘let’s try a green button.’ If it loses, you’ve learned nothing — not even why you tried. A strong hypothesis names the belief being tested, so either outcome updates your model of the customer.
I force the format: ‘Because [research insight], we believe [change] will cause [effect] on [metric], for [audience].’ Now a loss doesn’t just kill a variant — it disproves the belief in the ‘because,’ which is the valuable part.
The test isn’t really testing the button; it’s testing the assumption behind it. Write it so the assumption is what wins or loses.
The structured hypothesis
A structured hypothesis has four parts: the because (the research insight motivating it), the change (the specific intervention), the predicted effect (direction and rough size) on a named metric, and the audience it applies to. ‘Because session recordings show users miss the pricing toggle, adding an annual/monthly switch above the fold will increase paid-plan starts among new visitors.’ That sentence is testable, falsifiable, and tied to evidence — everything a guess is not.
The discipline of writing it out does real work: it forces you to have a reason (the ‘because’), to pick one effect and metric (not a vague ‘improve things’), and to scope the audience. Teams that skip this end up with tests they can’t interpret, because they never said what they expected or why — so any result can be rationalized after the fact.
A test’s trustworthiness is decided before it launches, not after. No end-of-test analysis can rescue a sloppy design.
One primary metric, plus guardrails
Choose one primary metric (the Overall Evaluation Criterion) that the test is designed to move, and commit to it before launch. Then add guardrail metrics — things that must not get worse (page speed, revenue per session, refunds, downstream retention). The primary metric decides the win; the guardrails catch wins that are secretly losses, like a checkout change that lifts conversions but tanks average order value or inflates returns.
Claim: Best practice defines a single Overall Evaluation Criterion (OEC) per experiment plus guardrail metrics, chosen in advance, to prevent cherry-picking the metric that happened to move. Source: Kohavi et al. — OEC and guardrail metrics. Context: Pre-committing one primary metric and guardrails is the cleanest defense against reading whatever number looks good after the fact.
Run a test with ten metrics and no declared primary, and something will always look ‘significant’ — you’ll find a winner by accident and ship it. That’s not analysis; it’s the multiple-comparisons trap dressed up as success.
So before launch I write down the one metric that decides the test and the guardrails that can veto it — and I treat everything else as exploratory color, not grounds to declare victory.
Deciding the scoreboard before kickoff is what stops a flat test from being retconned into a win on whatever number wandered upward.
Variant design
Design variants to isolate the variable you’re testing. If you change five things at once and the variant wins, you don’t know which change caused it — or whether one helped while three hurt. Either test one clean change (to learn what works) or test a bold, coherent redesign (to capture a big swing), but be clear which you’re doing: the first buys knowledge, the second buys impact, and confusing them gives you neither a clear lesson nor a clean win.
This is the classic tension between isolated and bundled tests. Isolated changes teach you precisely what moved the needle but win in small increments; bundled redesigns can deliver large lifts but leave you unsure why. Mature programs use both deliberately — bold bundles to find big territory, then isolated follow-ups to learn which elements drove it — rather than accidentally bundling and forfeiting the learning.
Randomization units
Decide what gets randomized — usually the user (consistent experience across visits), sometimes the session or device, occasionally a cluster (a whole account or geo). The unit must match the metric and avoid contamination: randomizing by session when you measure a multi-visit conversion lets the same person see both variants, polluting the result. Pick the unit so each subject experiences exactly one variant for the duration of what you’re measuring.
Randomization is the quiet foundation of causal validity, and it’s where subtle bugs hide. Cross-device users seeing different variants, logged-out vs logged-in inconsistency, or shared-account contamination all corrupt the comparison. For B2B or account-level effects, you often must randomize the whole account (cluster), which costs statistical power but prevents teammates from seeing different experiences — a trade you make on purpose.
Audience and targeting
Define who’s in the test and who’s excluded, in advance. Testing on the right audience (e.g. new visitors for an onboarding change, not your whole logged-in base) sharpens the signal and the applicability of the result. Beware over-narrowing — too small an audience can’t reach significance — and beware mixing populations whose behavior differs so much that the average hides opposing effects. Audience choice is part of the hypothesis, not an afterthought.
Duration and timing
Plan duration from the sample size needed to detect a meaningful effect, then run at least one to two full business cycles (usually whole weeks) to absorb day-of-week and buying-cycle variation. Don’t stop when it ‘looks significant’ — that’s peeking, and it manufactures false wins. Use a sample-size calculator and a duration estimator to set the end date before launch, and hold to it.
The single most common way good tests go bad: someone sees the variant ahead on day three, calls it, and ships. Early results swing wildly, and stopping on a high is how flat tests get shipped as winners.
I compute the required sample size and run-time up front, write the end date down, and treat it as non-negotiable — the test ends when it’s powered and a full cycle has passed, not when the line looks good.
If you must look early, look only to catch a disaster (a guardrail cratering), never to declare a win.
QA before you launch
Pre-launch QA is non-negotiable: confirm the variant renders correctly across browsers and devices, the tracking fires for the primary and guardrail metrics, the randomization actually splits traffic as intended, and there’s no flicker (the original flashing before the variant loads). A test that’s broken for 20% of users, or whose conversion event doesn’t fire, produces confidently wrong data — the worst kind. QA the experiment like you’d QA a deploy, because it is one.
- What is flicker and why does it matter?
- Flicker (FOOC — flash of original content) is the brief moment the original shows before the variant renders, caused by client-side testing tools. It biases results and hurts UX; minimize it with proper script placement or server-side/edge testing.
- How do I know randomization is working?
- Check for sample ratio mismatch (SRM) — the actual traffic split should match the intended split (e.g. 50/50). A significant deviation signals a bug that invalidates the test (covered in Module 3).
- Should I QA tracking before launch?
- Always. Fire the primary and guardrail events in each variant and confirm they record correctly before traffic flows. A test whose conversion event doesn’t fire in the variant will look like a catastrophic loss that isn’t real.
Where test design fails
Design failures: vague hypotheses with no ‘because,’ no pre-committed primary metric (so you cherry-pick), accidentally bundled variants (so you can’t learn), wrong randomization unit (contamination), audiences too small or too mixed, stopping early on peeking, and skipping QA. Each one corrupts the result before a single statistic is computed.
No ‘because’ means a loss teaches nothing and a win can’t be explained.
Watching many metrics guarantees a false ‘winner’ via multiple comparisons.
Changing five things at once forfeits the learning of what worked.
Session-level randomization on a multi-visit metric lets users see both variants.
Stopping early on noise and shipping broken variants both produce false results.
Your test-design checklist
Trustworthy tests are designed, not hoped for. Tick what is genuinely true of your test plan before launch.
Prove it. Earn your passcode.
Ten questions, CASE method (Context · Analysis · Strategy · Execution). Pass at 90% to unlock this module’s completion passcode — retake as many times as you like.