~/learn/A/B Testing · Hypothesis testing & inference
Non-parametric tests & bootstrap
Mann-Whitney U, bootstrapped confidence intervals and ratio metrics.
why this matters
Medians, percentiles and ratio metrics need more than a textbook t-test — strong candidates know the alternatives.
When assumptions are shaky — small samples, heavy tails, medians or percentiles — use other tools. Mann-Whitney U compares ranks rather than values, so it is robust to outliers; but it tests whether one distribution tends to be larger, not whether the means differ.
The bootstrap resamples users with replacement thousands of times, recomputes the metric difference each time, and uses the spread of those results as a confidence interval. It works for almost any statistic: medians, p95 latency, ratios.
Bootstrap loop
data → resample users (with replacement) → compute Δ
→ repeat 10,000× → take 2.5% and 97.5% percentiles = 95% CIRatio metrics
Ratio metrics like click-through rate per page view (clicks ÷ views, randomized by user) have rows that aren't independent. Use the delta method or bootstrap by user, never a naive per-view test.
key takeaways
- Mann-Whitney compares ranks, not means.
- Bootstrap works for almost any statistic.
- Ratio metrics need delta method or user-level bootstrap.
check your understanding
You want a 95% CI for the change in p95 page latency. Best approach?