~/learn/A/B Testing · Decisions & modern techniques

Making the ship decision

Conflicting metrics, practical vs statistical significance, and testing many things at once.

5 min readlesson 14 of 15tradeoffspractical significancemultiple testing

why this matters

Senior interviews end with 'so, would you ship it?'. A structured, numbers-first answer wins.

Statistical significance says an effect is probably real; practical significance says it's worth it. A +0.1% lift that is significant on 50 million users may not cover engineering and maintenance costs. Set a practical threshold before launch.

When metrics conflict — primary up, guardrail down — quantify the tradeoff in a common currency (revenue, retention), check whether the guardrail drop is real and how big, and escalate the call with clear numbers rather than hiding it.

Decision tree

SRM ok? ── no ──→ debug, don't ship
   │ yes
guardrails ok? ── no ──→ quantify tradeoff / iterate
   │ yes
primary sig. & ≥ practical threshold? ── yes → ship
   │ no
inconclusive → iterate or stop

Multiple testing

Testing 20 metrics or 20 segments at α = 5% gives about one false 'win' by chance. Correct for multiple comparisons (Bonferroni, Benjamini-Hochberg) or treat secondary results as exploratory.

key takeaways

Significant isn't the same as worth it.
Quantify guardrail tradeoffs.
Correct for multiple comparisons.

check your understanding

You tested 20 segments and one shows a significant win at α = 0.05. What's the risk?

Novelty effects & Simpson's paradox CUPED & variance reduction