~/learn/Product Metrics · Funnels & cannibalization
Simpson's paradox in product data
How a shift in user mix can make every segment improve while the total gets worse.
why this matters
Mix shift is the hidden cause behind many 'confusing' metric movements in case rounds.
Simpson's paradox: a trend appears in every subgroup but reverses when the groups are combined. In product data it is almost always caused by a change in mix — the share of users in each segment changed.
Example: conversion improves on both desktop and mobile, but overall conversion falls. Why? A big marketing push brought many more mobile users, and mobile converts at a lower rate. Each segment got better; the blend got worse.
Before After Desktop 500/5,000 = 10% 330/3,000 = 11% Mobile 100/5,000 = 2% 210/7,000 = 3% Total 600/10,000 = 6.0% 540/10,000 = 5.4% ← fell!
How to spot it
total moved? → check each segment → check segment sizes (mix) segments flat/up + total down → mix shift
Interview trap
When a top-line metric changes, always ask 'did the user mix change?' before blaming the product. It's one of the first things a strong candidate checks.
key takeaways
- Check every segment when totals move.
- Check whether segment sizes changed.
- Segments up + total down = mix shift.
check your understanding
Conversion rose on desktop and mobile, but fell overall. Most likely cause?