~/learn/A/B Testing · Sample size, power & duration
Duration & seasonality
Run full weekly cycles, avoid holiday distortion, and know when long tests hurt.
why this matters
Stopping on day 3 or running through Black Friday are classic mistakes interviewers probe for.
Duration = required sample size ÷ eligible daily traffic per group, rounded up to whole weeks. Behaviour differs by weekday — weekend shoppers are not weekday shoppers — so run at least one full week, typically two.
Avoid unusual periods (Black Friday, a big marketing push) unless the change is for that period. Very long tests have their own problems: cookies get cleared, users drift between groups, and the opportunity cost grows.
Whole weeks
Mon Tue Wed Thu Fri Sat Sun │ Mon Tue Wed Thu Fri Sat Sun └──────── week 1 ─────────┘ └──────── week 2 ─────────┘ stop at a week boundary, not mid-week
Pro tip
Ramp exposure gradually (1% → 10% → 50%) to catch bugs early, but only analyse the period when the split was stable.
key takeaways
- Run whole weeks — usually two.
- Avoid unusual periods.
- Analyse only the stable allocation period.
check your understanding
Your sample size is reached after 4 days. What should you do?