~/learn/A/B Testing · Hypothesis testing & inference
Two-proportion z-test
Testing conversion-rate differences, reading p-values and confidence intervals.
why this matters
Conversion tests are the bread and butter of A/B testing. Be ready to compute and interpret one by hand.
For binary outcomes (converted or not), compare conversion rates with a two-proportion z-test. Under the null of no difference, the observed difference divided by its standard error is roughly standard normal.
p̂ = pooled rate = (x_A + x_B) / (n_A + n_B) SE = √( p̂(1 − p̂) · (1/n_A + 1/n_B) ) z = (p_B − p_A) / SE |z| > 1.96 → significant at 5% 95% CI = (p_B − p_A) ± 1.96 · SE_unpooled
Report the confidence interval, not just the p-value. 'Lift +0.8pp, 95% CI [+0.2pp, +1.4pp]' tells the reader both the direction and the plausible size of the effect.
Read the CI
CI entirely above 0 → significant positive CI crosses 0 → inconclusive CI entirely below 0 → significant negative
Interview trap
Relative and absolute lifts are different. 10% → 11% is +1 percentage point absolute but +10% relative. Say which one you mean.
key takeaways
- z = difference ÷ standard error.
- Report confidence intervals, not just p-values.
- Distinguish absolute and relative lift.
check your understanding
The 95% CI for lift is [−0.3pp, +1.1pp]. What do you conclude?