~/learn/A/B Testing · Sample size, power & duration
MDE & sample size
The minimum detectable effect, the sample-size formula, and why small effects are expensive.
why this matters
'How many users do we need?' is a standard question. Showing the inverse-square relationship signals real understanding.
The minimum detectable effect (MDE) is the smallest lift your test can reliably detect at your chosen α and power. You pick it before launch based on what is worth shipping. Sample size depends on the metric's variance and the MDE.
n per group ≈ 16 · σ² / δ² (α = 0.05, power = 80%) conversion metric: σ² = p(1 − p) δ = absolute MDE (e.g. 0.5 percentage points)
Because δ is squared, halving the MDE needs four times the users. Revenue-type metrics have huge variance (most users spend nothing, a few spend a lot), so they need far more traffic than a simple conversion rate.
Inverse square
MDE 2% → n MDE 1% → 4n MDE 0.5% → 16n
Interview trap
Don't choose the MDE from the traffic you happen to have and call the test 'powered'. If the realistic effect is smaller than your MDE, the test will most likely be inconclusive.
key takeaways
- n ≈ 16σ²/δ² per group.
- Halving the MDE → 4× the users.
- High-variance metrics need more traffic.
check your understanding
You need 10,000 users per group to detect a 2pp lift. Roughly how many to detect 1pp?