~/learn/ML System Design · Serving & latency
Online vs batch inference
Predict on request, or precompute and look up? The core serving decision.
why this matters
Batch versus real-time is the first serving decision in nearly every design.
Batch (offline) inference: run the model on everyone on a schedule and store the results — e.g. nightly 'recommended for you' lists or churn scores in a table. Cheap, simple, and fast to serve (just a lookup), but stale and wasteful for users who never return.
Online (real-time) inference: run the model when the request arrives, using the freshest context (current search query, session clicks, basket). Required for search, fraud checks at payment time, and ride pricing.
Three options
batch model ─► precomputed table ─► lookup (stale, cheap) online request ─► features ─► model ─► answer (fresh, latency budget) hybrid precomputed candidates + real-time re-ranking
Most production systems are hybrid: generate candidates or embeddings in batch, then score or re-rank online with fresh signals.
Interview trap
Choosing online inference by default. If inputs don't change during the day (e.g. a weekly email), batch is simpler and cheaper — say why.
Pro tip
Always design a fallback: if the model times out, return popular items or the cached result rather than an error.
key takeaways
- Batch: precompute and look up.
- Online: fresh context, latency budget.
- Hybrid is common; always have a fallback.
check your understanding
A weekly 'recommended products' email is best served by: