~/learn/A/B Testing · Diagnostics & classic pitfalls

Interference & network effects

When treatment leaks to control: marketplaces, social features and switchback tests.

5 min readlesson 12 of 15interferencemarketplaceswitchback

why this matters

Marketplace and social companies (Uber, Airbnb, Meta, DoorDash) almost always ask about interference.

Standard A/B tests assume one user's group doesn't affect another's outcome (SUTVA). Marketplaces and social products break this. If treatment riders get a discount and book more Ubers, fewer drivers are left for control riders — control looks worse and the lift is overstated.

Social features leak too: if treatment users can share a new sticker, their control friends see it. Fixes: randomize by cluster (city, friend-graph community), or use switchback tests that flip the entire market between A and B over time windows.

Spillover

rider-level test:  treatment ↑ demand → steals drivers → control ↓
                   measured lift = true lift + cannibalisation
switchback:  [A 1h][B 1h][A 1h][B 1h] ... whole city at once

Interview trap

Cluster randomization has far fewer units (dozens of cities, not millions of users), so power drops. Account for it in sample sizing.

key takeaways

Spillover violates independence.
Use cluster randomization or switchbacks.
Fewer units means less power.

check your understanding

Testing a rider discount in a ride-sharing app, user-level randomization will likely…

Peeking & early stopping Novelty effects & Simpson's paradox