P-value
It is NOT the chance your result is real — and that single misreading has wrecked more decisions than any other in analytics.
- Term
- P-value
- Measures
- Surprise under the null hypothesis
- Common threshold
- p < 0.05
- Is NOT
- The probability the effect is real
Forms & parts of speech
Definition in plain terms
A p-value is the probability of observing a result at least as extreme as the one you got, ASSUMING there is no real effect (the null hypothesis is true). A p-value of 0.03 means 'if nothing were actually happening, you'd see a result this strong only 3% of the time' — surprising enough, by the p < 0.05 convention, to call the result statistically significant. Crucially, it does NOT say the effect is 97% likely to be real; that's the single most damaging misreading in applied analytics.
The mechanics
The p-value answers a narrow question (how surprising is this data under 'no effect?') and is routinely asked to answer a different one (how likely is my hypothesis true?) — the inversion fallacy. Its real-world failure modes: P-HACKING (testing many variants or metrics until one crosses 0.05 — with 20 tests, one false positive is expected by chance), PEEKING (checking repeatedly and stopping when significant — which inflates the false-positive rate far above 5%), and confusing statistical significance with PRACTICAL significance (a real but trivially small effect can be 'significant' at huge sample sizes). The American Statistical Association's 2016 statement formally warned against exactly these abuses.
When it matters
The p-value matters as one input to a decision, never the decision itself — read alongside the effect SIZE (is it big enough to matter?), the confidence interval (how precise?), prior plausibility, and whether the test was pre-registered or fished. For marketers the discipline is cultural: a team that ships on 'p < 0.05' alone will ship noise, while a team that asks 'how big, how precise, how surprising, and did we go looking for it?' makes durable decisions. Significance is a smoke alarm, not a verdict.
Synonyms & antonyms
Synonyms
Antonyms
Origin & history
The p-value was popularized by Ronald Fisher in Statistical Methods for Research Workers (1925), where he proposed the 5% threshold as a convenient (not sacred) convention; the Neyman-Pearson framework added the formal error-rate machinery, and the ASA's 2016 statement codified the modern warnings against its misuse.
Etymology: source.
Usage trends
Search interest for this term over the last five years:
Common questions
- What is a p-value?
- The probability of seeing a result at least as extreme as yours if there were no real effect — a measure of surprise under the null.
- What does a p-value NOT tell you?
- It is not the probability your hypothesis is true, nor the probability the effect is real — that inversion is the common fatal error.
- What is p-hacking?
- Testing many variants or metrics until one crosses the significance threshold — manufacturing false positives by chance.
Related tools & calculators
Resources & people to follow
- referenceASA Statement on p-values (2016)
- bookTrustworthy Online Controlled Experiments — Kohavi, Tang & Xu
- referenceRGM analysis — read size, precision, and intent alongside p
Curated, non-competitor resources verified per term.
Related training
- moduleCRO & experimentation
Disciplines
Areas of marketing where p-value is a core concern: