Trustworthy Online Controlled Experiments
The single book to read if you run experiments for a living. Dense in the best way, and honest about how easy it is to fool yourself.
What it is
Written by three people who built and ran experimentation platforms at Microsoft, Amazon, LinkedIn and Airbnb, this is the closest thing the field has to a canonical text. It moves from the philosophy of controlled experiments to the gritty mechanics of running them at scale - randomization, the statistics of the two-proportion test, sample size and power, and the institutional habits that keep an experimentation program honest.
The book is organized so you can read the foundational chapters cover to cover and then treat the advanced sections as a reference. That structure matters, because the second half is where the real value sits.
What's strong
The authors are relentless about the ways experiments lie. Twyman's Law - 'any figure that looks interesting or different is usually wrong' - runs through the book, and they back it with concrete failure modes: the novelty effect, sample-ratio mismatch, the peeking problem, and the temptation to declare a breakthrough on an underpowered test. Their famous example - a Bing ad-headline change that few believed in, tested, and worth roughly 100 million dollars a year - is the clearest argument you will find for testing the boring ideas too.
It is also refreshingly practical about organizational reality: how to build trust in results, how to avoid HiPPO-driven decisions, and why most ideas simply do not move the metric they were meant to move.
Where it stops
This is not a gentle introduction. If you have never run a test, the statistics chapters will be heavy going, and the book assumes you have data infrastructure or the ambition to build it. Bayesian approaches are covered, but the authors are firmly in the frequentist, fixed-horizon tradition; readers committed to sequential or Bayesian methods will want to supplement it. None of that is a flaw - it is a book for practitioners who are ready to take the work seriously.
Who should read it
Anyone responsible for an experimentation program, growth or product analysts who size and call tests, and engineers building test infrastructure. If you have ever argued about whether a result was 'significant yet,' this book settles the argument with rigor. Pair it with our A/B test budget calculator before you scope your next test.
How RGM uses it
We treat this book as the house standard for what 'trustworthy' means before we report a test result to a client. The chapters on trust, validity checks, and sample-ratio mismatch shape our pre-flight checklist: we confirm randomization held, we refuse to call winners on underpowered reads, and we treat any surprisingly large lift as a flag to investigate rather than a reason to celebrate. The 100-million-dollar Bing example is one we cite often when a client wants to skip 'small' ideas - it is the cleanest evidence that the size of an idea on paper tells you almost nothing about its value once tested. If you read one experimentation book this decade, read this one, then keep it within reach.