~/learn/ML System Design · Data & feature pipelines
Feature stores & training-serving skew
One definition of each feature, served offline for training and online for inference.
why this matters
Training-serving skew is a famous silent failure; feature stores are the standard fix.
Training-serving skew happens when a feature is computed one way in the training pipeline (SQL in the warehouse) and a slightly different way in production (Java in the service). The model sees different inputs in production than it learned from, and quality silently drops.
A feature store (Feast, Tecton, internal tools at Uber and Airbnb) fixes this by defining each feature once and materializing it into two places: an offline store (warehouse) for training data, and an online store (Redis, DynamoDB) for millisecond lookups at serving time.
Dual storage
feature definition (once)
├──► offline store (warehouse) → build training sets
└──► online store (key-value) → < 10 ms lookups at servingPoint-in-time correctness: when building training data, each row must use feature values as they were at that event's timestamp — not today's values. Feature stores do these 'as-of' joins for you.
Interview trap
Joining today's feature values onto last year's labels is leakage — the model learns from information it won't have in production.
Pro tip
A cheap skew safeguard: log the exact features used at serving time and train the next model on those logs.
key takeaways
- Define each feature once.
- Offline store for training, online store for serving.
- Use point-in-time joins.
check your understanding
What is training-serving skew?