Replace opinion with evidence — and run the program, not the test
Most ‘optimization’ is redesigning and hoping. Experimentation is different: a controlled test tells you a change actually caused the result, so every outcome — win, loss, or flat — banks a durable fact about your customers. The catch is that roughly two-thirds of good ideas won’t win, so the value isn’t any single test — it’s the compounding knowledge of a program run with discipline.
So this course is built around running the program, not chasing the test. You’ll learn to choose high-leverage tests (Module 1), design and power them so the result is trustworthy (2–3), prioritize ruthlessly when traffic is scarce (4), reach for bandits and Bayesian methods when earning-while-learning matters (5), and build the culture where evidence beats the highest-paid opinion in the room (6). Every module ends with a hard CASE-method test, and the calculators it links are the ones we use on client work.
Getting numbers is easy; getting numbers you can trust is hard.
— Ronny Kohavi, Trustworthy Online Controlled Experiments
Working calculators for this series: A/B test sample size, statistical significance, test duration, ICE score, RICE score, and the experiment planner.
Four steps to your certification
No sign-in, no account. Take the modules in any order, pass the CASE-method tests, and collect a passcode for each. Clear all six plus the final exam and you’re certified.
Learn the module
Deep, practical lessons with decoders, linked calculators, and sourced research — no fluff.
Pass the test
Hard, scenario-based CASE-method questions. Score 90% to clear the module.
Earn a passcode
Each pass reveals a secret code. Save it — re-enter any time to reclaim credit and jump ahead.
Get certified
Clear all six modules and the final exam to unlock your certificate.
Six modules
Fundamentals first, then the design and statistics that keep tests honest, then prioritization, then bandits and Bayesian methods, then the culture that decides it all. Each module ends with a hard, subject-specific test; each pass unlocks a passcode.