~/learn/Machine Learning · Production ML & modern concepts
ML systems & monitoring
Data drift vs concept drift, offline vs online evaluation, and training-serving skew.
why this matters
Senior ML interviews focus on what happens after launch: drift, skew and proving impact with experiments.
Data drift (covariate shift) is when the input distribution changes — new markets, a new app version. Concept drift is when the relationship between inputs and the label changes — fraud patterns evolve, user tastes shift. Data drift can be seen without labels; concept drift usually needs fresh labels to detect.
Offline metrics (AUC on a holdout) don't guarantee business impact. Online evaluation — A/B tests, interleaving, shadow deployments — measures what actually happens. Training-serving skew appears when features are computed differently in training and in production; feature stores help by sharing one definition.
The feedback loop
data ─▶ features ─▶ model ─▶ predictions ─▶ outcomes ▲ monitor inputs ▲ monitor scores ▲ monitor labels/KPIs └──────────── retrain when drift is detected ─────────┘
Interview trap
Feedback loops: a recommender only learns from items it chose to show. Without exploration, it reinforces its own biases.
Pro tip
Monitor input distributions (PSI, KS tests), prediction distributions and delayed-label performance — and alert on each.
key takeaways
- Data drift = inputs change; concept drift = relationship changes.
- Validate impact online with A/B tests.
- Watch for training-serving skew and feedback loops.
check your understanding
Users' input features look unchanged, but model accuracy is dropping as fraudsters adapt. This is…