~/learn/ML System Design · Serving & latency
Safe deployment & rollouts
Shadow mode, canaries and A/B tests — never flip 100% of traffic at once.
why this matters
Showing you can ship safely signals production experience.
A new model can be offline-better and still break production (slower, crashes on rare inputs, odd behaviour). So rollouts are staged.
Rollout ladder
1. offline eval beats baseline on held-out data 2. shadow mode runs on live traffic, results not shown 3. canary 1–5% of traffic, watch errors & latency 4. A/B test measure business metrics vs control 5. full rollout keep old model ready for rollback
Shadow mode compares the new model's predictions with the old one's on real inputs, with zero user risk. Canary releases catch crashes and latency spikes on a small slice. The A/B test decides whether it's actually better for users.
Keep a model registry: every model version, its training data snapshot, metrics and config, so you can reproduce or roll back instantly.
Interview trap
Skipping the A/B test because offline metrics improved. Offline wins frequently don't hold online.
Pro tip
Watch for prediction distribution shifts during canary: if the new model's average score is wildly different, something upstream is probably broken.
key takeaways
- Shadow → canary → A/B → full.
- Keep a model registry.
- Always be able to roll back.
check your understanding
What is shadow mode?