~/learn/Product Metrics · Diagnosing a metric drop
Framework: diagnosing a metric drop
A structured, step-by-step approach to 'metric X dropped 10% — what happened?'
why this matters
'Metric X dropped — what happened?' is the most frequent product analytics interview question.
'Daily active users dropped 10% last week. How would you investigate?' is the single most common product analytics case question. Interviewers are testing structure, not whether you guess the answer. Work through it in order and say each step out loud.
The 7-step framework
1. Clarify — which metric, exact definition, how much, since when? 2. Validate — is the data real? logging, pipelines, dashboards 3. Scope — sudden or gradual? one-off or trend? seasonal? 4. Segment — platform, country, version, new vs old, source 5. Decompose — break the metric into its parts (metric tree) 6. Hypothesise — internal (launch, bug) vs external (holiday, competitor) 7. Confirm & act — test the top hypothesis, size impact, recommend
Clarify first: a 10% drop day-over-day on a Sunday may be completely normal. Then rule out data issues — a surprising number of real drops are broken tracking. Segmenting usually finds the answer: if the whole drop is on Android version 5.2, you're looking at a bad release.
DAU = new users + retained users + resurrected users → which component fell?
Interview trap
Jumping straight to 'maybe a competitor launched something' without clarifying or checking data quality is a red flag. Start with the boring checks.
Pro tip
Close by sizing and recommending: 'The drop is fully explained by the Android 5.2 crash, ~120k DAU. I'd roll back, add a crash-rate alert, and re-check in 48 hours.'
key takeaways
- Clarify, then validate the data.
- Segment and decompose to localize.
- Hypothesise, confirm, recommend.
check your understanding
DAU dropped 10% overnight. What should you do first after clarifying the definition?