Statistical Power
A test can be 'not significant' because the effect isn't real — or because the test was too weak to see it. Power is the difference.
- Term
- Statistical Power
- Definition
- 1 − β (the false-negative rate)
- Convention
- 80% target (90%+ for high stakes)
- Set before
- Run — power is a design choice
Forms & parts of speech
Definition in plain terms
Statistical power is the probability that an experiment correctly detects a real effect WHEN ONE TRULY EXISTS — formally, 1 minus the false-negative (type II error) rate. A test with 80% power has an 80% chance of finding a genuine effect of the size you care about, and a 20% chance of missing it. Power is the quiet, decisive half of experiment design that significance testing's fame obscures: a 'not significant' result from an underpowered test tells you almost nothing.
The mechanics
Power is determined before the test by four interlocked quantities: SAMPLE SIZE (more power), the MINIMUM DETECTABLE EFFECT you want to catch (smaller effects need more power), the significance threshold (alpha), and the metric's variance. Fix three and the fourth follows — which is why power analysis is really sample-size planning. The convention is 80% power at 5% significance, but high-stakes decisions warrant 90%+. The trap underpowered tests set: they generate a stream of 'inconclusive' results that get misread as 'no effect,' killing good ideas and wasting the traffic spent testing them.
When it matters
Power matters BEFORE every test — calculating required sample size from your baseline rate and the smallest worthwhile effect is the entry fee for honest experimentation. It matters most for low-traffic pages and small effects (where adequate power may require months, or the test simply isn't worth running). For marketers it reframes the testing roadmap: power analysis tells you which experiments your traffic can actually answer, so you stop running tests that were doomed to be inconclusive before they began.
Synonyms & antonyms
Synonyms
Antonyms
Origin & history
The concept is foundational to the Neyman-Pearson framework of hypothesis testing (1933), which formalized type I and type II errors; Jacob Cohen's Statistical Power Analysis for the Behavioral Sciences (1969) made power analysis a standard research practice and gave the field its 80% convention.
Etymology: source.
Usage trends
Search interest for this term over the last five years:
Common questions
- What is statistical power?
- The probability a test detects a real effect when one exists — conventionally targeted at 80% or more.
- Why does power matter?
- An underpowered test's 'no significant difference' often means 'too weak to tell,' not 'no effect' — wasting traffic and killing good ideas.
- What determines power?
- Sample size, the minimum detectable effect, the significance threshold, and the metric's variance — set before the test runs.
Related tools & calculators
Resources & people to follow
- bookTrustworthy Online Controlled Experiments — Kohavi, Tang & Xu
- referenceCohen — statistical power analysis foundations
- referenceRGM analysis — power analysis decides which tests your traffic can answer
Curated, non-competitor resources verified per term.
Related training
- moduleCRO & experimentation
Disciplines
Areas of marketing where statistical power is a core concern: