~/learn/A/B Testing · Diagnostics & classic pitfalls
Sample ratio mismatch (SRM)
The first health check: is the traffic split what you designed?
why this matters
Asking about SRM separates people who have run real experiments from people who read about them.
If you designed a 50/50 split and got 50,000 vs 48,500 users, something is wrong. A sample ratio mismatch means assignment or logging is biased, so the groups are no longer comparable and the results can't be trusted — no matter how good they look.
χ² = Σ (observed − expected)² / expected (1 degree of freedom) flag SRM if p < 0.001
Common causes: the treatment page loads slower so more users bounce before being logged, bots filtered differently, a redirect in only one variant, or a bug that excludes some users from one group.
Health first
1. check SRM → 2. check guardrails → 3. read the primary metric (fail at 1 = stop and debug, don't analyse)
Interview trap
Never 'fix' an SRM by down-sampling the bigger group. The bias is in who went missing, not in the counts.
key takeaways
- Check the split with a chi-square test.
- SRM means biased groups — don't trust results.
- Debug the cause instead of rebalancing.
check your understanding
A 50/50 test shows 100,000 vs 97,000 users (p < 0.0001). What now?