~/learn/ML System Design · Data & feature pipelines
Training data, labels & sampling
Where labels come from, how to build negatives, and how leakage sneaks in.
why this matters
Bad labels and leakage sink more real projects than model choice does.
Labels come from three places. Explicit feedback: ratings, thumbs up, reports. Implicit feedback: clicks, purchases, watch time — plentiful but noisy. Human annotation: expensive but high quality, used for content moderation and safety.
For recommendations you only see what users did with items they were shown. Unclicked impressions are your negatives. Items never shown are not true negatives — the user simply never saw them.
Class imbalance is normal: fraud may be 0.1% of transactions. Common fixes: downsample the majority class (and recalibrate predicted probabilities afterwards), upweight the rare class, or pick metrics like PR-AUC that ignore easy negatives.
Implicit labels
impression logs ─┬─ clicked → positive
├─ shown, skipped → negative
└─ never shown → unknown (not a negative!)Interview trap
Data leakage: a feature that wouldn't be known at prediction time (e.g. 'refund issued' used to predict fraud). Always split train/test by time, not randomly, so the model never trains on the future.
Position bias
Position bias: items at the top get clicked more just because they're at the top. Log the position and add it as a feature during training (set to a constant at serving).
key takeaways
- Implicit feedback is plentiful but noisy.
- Unseen items aren't true negatives.
- Split by time to avoid leakage.
check your understanding
Which split best avoids leakage for a fraud model?