~/learn/Machine Learning · Feature engineering
Categorical encodings
One-hot, ordinal, target encoding and how to avoid leakage with high cardinality.
why this matters
Real datasets are full of IDs, cities and product codes. How you encode them is a frequent take-home discussion point.
One-hot encoding creates a binary column per category — safe, but explodes with high cardinality (e.g. 50,000 zip codes). Ordinal encoding maps categories to integers and only makes sense when there is a natural order, or with tree models.
Target encoding replaces a category with the average target for that category. It's compact and powerful, but leaks the label unless computed out-of-fold, and it overfits rare categories unless smoothed toward the global mean.
smoothed = (n · mean_category + m · mean_global) / (n + m)
Interview trap
Plan for categories that appear only at prediction time — add an 'unknown' bucket or fall back to the global mean.
Pro tip
Alternatives for high cardinality: frequency encoding, hashing, or learned embeddings.
key takeaways
- One-hot is safe for low cardinality.
- Target encoding must be out-of-fold and smoothed.
- Plan for unseen categories.
check your understanding
Why compute target encoding out-of-fold?