~/learn/A/B Testing · Sample size, power & duration
Type I & II errors and power
False positives, false negatives, significance level α and statistical power.
why this matters
α, β and power are the vocabulary of every experimentation interview — get them precise.
Two mistakes are possible. A Type I error (false positive) ships a change that did nothing; its rate is α, usually 5%. A Type II error (false negative) misses a change that really worked; its rate is β, usually 20%.
Power = 1 − β: the chance you detect a real effect of a given size. 80% power means that if the true lift equals your target effect, 8 out of 10 runs of this experiment would show a significant result.
Error matrix
truth: no effect truth: real effect significant Type I (α ≈ 5%) correct (power ≈ 80%) not significant correct Type II (β ≈ 20%)
Interview trap
A p-value is NOT the probability that the null is true. It's the probability of seeing data at least this extreme if the null were true.
Absence of evidence
'Not significant' doesn't mean 'no effect' — an underpowered test simply couldn't see it.
key takeaways
- α = false positive rate (5%).
- Power = 1 − β, chance to detect a real effect (80%).
- Not significant ≠ no effect.
check your understanding
A test with 80% power means…