~/learn/ML System Design · Monitoring & feedback loops
Drift & monitoring
Models decay silently. Know what to watch and how to detect it.
why this matters
Models degrade silently; interviewers expect a concrete monitoring plan.
Software fails loudly; ML models fail quietly — they keep returning numbers, just worse ones. Monitoring is how you notice.
Data drift (covariate shift): the inputs change — new user demographics, a new app version, a holiday. Concept drift: the relationship between inputs and the label changes — fraudsters adapt, tastes shift. Upstream breakage: a pipeline starts sending nulls or a unit changes from dollars to cents.
Monitoring stack
layer what to monitor system latency, errors, throughput data missing values, ranges, feature distributions (PSI) predictions score distribution, share of positives business CTR, conversion, complaints (delayed labels)
Population Stability Index (PSI) compares a feature's current distribution to its training distribution: below 0.1 is stable, 0.1–0.25 worth a look, above 0.25 a significant shift. The Kolmogorov–Smirnov test is another common choice.
Interview trap
Labels often arrive late (a loan defaults months later), so you can't measure accuracy in real time. Monitor input and prediction distributions as early warning signals.
Pro tip
Most 'model problems' in production are actually data pipeline problems. Schema and null checks catch a surprising share of incidents.
key takeaways
- Data drift vs concept drift.
- Watch inputs and predictions when labels are delayed.
- PSI above 0.25 is a significant shift.
check your understanding
Fraudsters change tactics so old patterns no longer predict fraud. This is: