~/learn/A/B Testing · Decisions & modern techniques
CUPED & variance reduction
Use pre-experiment data to shrink noise and get answers in less time.
why this matters
CUPED is the most-asked advanced technique at companies with mature experimentation platforms.
Much of the variation in a metric comes from who users already are: heavy users spend more in both groups. CUPED (Controlled-experiment Using Pre-Experiment Data) subtracts the part of each user's metric that is predictable from their pre-experiment behaviour, leaving a less noisy metric with the same expected treatment effect.
Y_cuped = Y − θ · (X − mean(X)) θ = cov(X, Y) / var(X) X = same metric in the weeks before the test variance reduced by a factor of (1 − ρ²)
If pre- and post-period spend correlate at ρ = 0.7, variance falls by about 49% — roughly halving the sample size or test duration. It's used at Microsoft, Netflix, Booking and most mature experimentation platforms.
Interview trap
The covariate must be measured before assignment. Using anything the treatment could affect introduces bias.
Pro tip
New users have no history. Fill X with 0 plus a 'new user' flag, or use other pre-assignment covariates like country and platform (regression adjustment).
key takeaways
- Subtract what's predictable from pre-period data.
- Variance shrinks by (1 − ρ²).
- Covariates must be pre-assignment.
check your understanding
Pre-period and in-test spend correlate at ρ = 0.6. Roughly how much does CUPED cut variance?