~/learn/A/B Testing · Foundations & metric selection
The experimentation lifecycle
Why we randomize, how to write a testable hypothesis, and the steps from idea to decision.
why this matters
Almost every product-analytics interview opens with 'how would you test this feature?'. A crisp lifecycle answer sets the tone.
An A/B test answers a causal question: did this change cause the metric to move? Comparing users before and after a launch mixes your change with everything else that happened that week. Randomly splitting users into control (A, unchanged) and treatment (B, the change) builds a fair counterfactual — the only systematic difference between the groups is your change.
A good hypothesis is specific and falsifiable: 'If we shorten checkout from 3 steps to 1, purchase conversion will increase, because fewer steps means fewer drop-offs.' It names the change, the metric, the expected direction and the reason.
The lifecycle
idea → hypothesis → pick metrics → size the test
→ randomize & launch → check health (SRM)
→ analyse → decide → document & shareInterview trap
Writing the hypothesis after seeing the results is not a hypothesis — it's storytelling. Fix the primary metric and decision rule before launch.
Pro tip
Run an A/A test (both groups get the same experience) on a new platform. It should show 'no difference' about 95% of the time — if not, your pipeline is broken.
key takeaways
- Randomization creates a fair counterfactual.
- Write a falsifiable hypothesis before launch.
- Use A/A tests to validate your platform.
check your understanding
Why can't you just compare metrics before and after launching a feature?