~/learn/A/B Testing · Foundations & metric selection
Unit of randomization
User, session, device or cluster — and why the choice changes both experience and statistics.
why this matters
Choosing the wrong unit is a silent killer of results; it's a favourite follow-up question.
The randomization unit is what gets assigned to A or B. User-level (by logged-in ID) is the default: each person sees one consistent experience. Session- or page-level gives more samples but the same person can flip between versions, which is confusing and breaks for anything users remember (prices, layout).
Device or cookie IDs work for logged-out traffic but one person on two devices may land in both groups. Cluster-level randomization (by city, school, or seller) is used when users influence each other.
The tradeoff
finer unit (page, session) → more samples, less consistency coarser (user, device) → consistent experience coarsest (city, cluster) → handles interference, few samples
Interview trap
The analysis unit should match the randomization unit. Randomize by user but compute variance as if page views were independent, and your p-values will be far too small.
key takeaways
- User-level is the default.
- Session-level breaks consistent experiences.
- Analyse at the randomization unit.
check your understanding
You're testing a new pricing page. Which unit fits best?