~/learn/A/B Testing · Diagnostics & classic pitfalls
Peeking & early stopping
Why checking results every day inflates false positives, and how sequential tests fix it.
why this matters
'Can we stop early? It's already significant!' — every analyst hears this from stakeholders.
A fixed-horizon test assumes you look once, at the planned sample size. If you check the p-value daily and stop the first time it dips below 0.05, random noise will cross that line eventually. With daily peeks over a month, the false positive rate can climb from 5% to 20–30%.
Random walk
p-value
0.5 ┤ ╭╮ ╭─╮
│ ╭╯╰╮ ╭╯ ╰╮ ╭─╮
0.05 ┼─────╰─╯───╰──╯─╰── ← noise crosses by chance
└──────────────────── daysIf you need to monitor and stop early, use a method built for it: sequential testing (e.g. mSPRT, always-valid p-values) or group-sequential designs with spending functions (O'Brien-Fleming) that set stricter thresholds for early looks.
Peek ≠ stop
Looking at the dashboard is fine. Deciding to stop because it looks good is the problem.
Pro tip
It's fine — and good — to stop early for harm: if a guardrail like crash rate breaks badly, kill the test.
key takeaways
- Repeated peeking inflates false positives.
- Use sequential methods if you must stop early.
- Stopping for harm is fine.
check your understanding
A PM checks daily and stops as soon as p < 0.05. What happens to the false positive rate?