~/learn/A/B Testing · Decisions & modern techniques
Novelty effects & Simpson's paradox
Short-term excitement, change aversion, and aggregates that reverse within segments.
why this matters
'The lift was big in week 1 and shrank in week 2 — why?' is a common case question.
Novelty effect: users click a shiny new feature because it's new, and the lift fades. Primacy (change aversion): experienced users resist a new layout at first, and the effect improves over time. Plot the treatment effect by day, or by user tenure, and see whether it is stable.
Simpson's paradox: a trend that appears in every segment can reverse in the aggregate when group mixes differ. If treatment received a different share of mobile users (for example after a ramp change), the overall comparison can mislead.
Simpson's paradox
desktop mobile overall control 10% 2% 8% (mostly desktop) treat 11% 3% 5% (mostly mobile) → treatment wins in every segment but loses overall
Pro tip
Compare new vs returning users: if new users (who have no habits) show the same lift, the effect is less likely to be novelty.
key takeaways
- Novelty fades; change aversion recovers.
- Plot the effect over time and by tenure.
- Mix shifts can reverse aggregates.
check your understanding
A new feed layout shows +8% engagement in week 1, +2% in week 3. Most likely?