~/learn/ML System Design · Framework & problem framing
Offline vs online metrics
Model metrics, business metrics and system guardrails — and why they disagree.
why this matters
Knowing which metric decides what — and why they disagree — is a classic follow-up question.
Offline metrics are computed on held-out historical data before launch: AUC and log loss for classifiers, precision@K, recall@K and NDCG for rankings, MAE for regression. They are fast and cheap, so you use them to compare candidate models.
Online metrics are measured on real users in an A/B test: click-through rate, watch time, conversion, revenue, retention. These are what the business actually cares about.
System metrics are guardrails: P99 latency (the slowest 1% of requests), throughput (queries per second), cost per 1,000 predictions, and error rate.
Metric layers
offline (AUC, NDCG) → pick candidates, cheap, historical ▼ must correlate with online (CTR, retention) → the real decision, A/B test + guardrails → P99 latency, cost, errors
Interview trap
Offline gains often don't translate online: the historical data was produced by the old model (selection bias), and user behaviour changes when recommendations change. That's why every launch still needs an A/B test.
Pro tip
Quote latency as a percentile: 'P99 under 100 ms' — averages hide the slow requests users actually notice.
key takeaways
- Offline metrics compare models cheaply.
- Online A/B metrics make the launch decision.
- Guard P99 latency and cost.
check your understanding
Why can a model with better offline AUC lose an online A/B test?