~/learn/ML System Design · Serving & latency
Making models fast
Quantization, distillation, caching and batching to hit a latency budget.
why this matters
'Your model is too slow — what now?' is a guaranteed follow-up.
Start with a latency budget and divide it across stages: e.g. 200 ms total = 20 ms feature lookups + 10 ms retrieval + 60 ms ranking + overhead. Then optimize the stage that blows its budget.
Model-level tricks: quantization stores weights as 8-bit integers instead of 32-bit floats (≈4× smaller, 2–4× faster, tiny accuracy loss). Distillation trains a small 'student' model to copy a big 'teacher'. Pruning removes weights that contribute little.
System-level tricks: cache results for repeated requests (popular queries), batch many requests into one GPU call, use optimized runtimes (ONNX Runtime, TensorRT), run on GPUs only where it pays off.
Levers
lever typical gain cost quantization 2–4× faster small accuracy drop distillation 5–10× smaller extra training work caching ~0 ms on hits staleness batching higher throughput adds waiting time
Interview trap
Dynamic batching improves throughput but adds latency (requests wait to fill a batch). Name the tradeoff.
Pro tip
For LLM-powered features, mention caching embeddings/responses, streaming tokens and smaller models for easy queries (routing).
key takeaways
- Split the latency budget by stage.
- Quantize or distill the model.
- Cache and batch at the system level.
check your understanding
What does INT8 quantization do?