~/learn/A/B Testing · Hypothesis testing & inference
Welch's t-test for averages
Comparing means like revenue per user, handling skew, outliers and unequal variances.
why this matters
Revenue metrics are where naive analysis breaks — whales and skew come up constantly.
For continuous metrics (revenue per user, minutes watched) compare means with a t-test. Welch's version does not assume equal variances, which is the safe default because a treatment can change the spread as well as the mean.
t = (x̄_B − x̄_A) / √( s_A²/n_A + s_B²/n_B )
Thanks to the central limit theorem, the mean is approximately normal with large samples, even if individual spend is heavily skewed. But a few whales can still dominate variance and swing results.
Pro tip
Cap (winsorize) extreme values at e.g. the 99.9th percentile, decided before looking at results. It cuts variance sharply with little bias.
Transforms change the question
Log-transforming revenue changes the question: you're now comparing geometric means, and users with zero revenue need special handling. Make sure that's the decision you want.
key takeaways
- Welch's t-test doesn't assume equal variances.
- Cap outliers using a pre-set rule.
- Log transforms change what you measure.
check your understanding
One user spends $50,000 in treatment and flips the result. The best pre-planned fix is…