~/learn/A/B Testing · Foundations & metric selection
Metrics: primary, secondary & guardrails
Choose a sensitive primary metric, protect the business with guardrails, and beware of proxies.
why this matters
Interviewers want to hear one primary metric, a reason for it, and the guardrails that stop you shipping something harmful.
The primary (or goal) metric decides the experiment — there should be one, chosen before launch. Secondary metrics explain why it moved (clicks, add-to-cart, time on page). Guardrail metrics must not get worse: page latency, crash rate, unsubscribes, refunds, support tickets.
A good primary metric is aligned with long-term value, sensitive enough to move within the test window, and hard to game. Long-term outcomes like 90-day retention are often too slow, so teams use a proxy (7-day active days) that is shown to predict the long-term metric.
Metric stack
North Star long-term value (revenue, retention) ▲ predicts Primary metric decides this test Secondary explain the movement Guardrails must not regress (latency, churn)
Interview trap
Click-through rate is easy to move and easy to game: clickbait raises CTR while hurting satisfaction. Always pair an engagement metric with a quality guardrail.
Pro tip
Define metrics per user (conversions per user) rather than per page view, so heavy users don't dominate and the unit matches randomization.
key takeaways
- One primary metric decides the test.
- Guardrails must not regress.
- Proxies should predict long-term value.
check your understanding
You're testing a bigger 'Buy' button. Which is the best guardrail?