~/learn/Machine Learning · Classical algorithms
Tree-based ensembles
Decision trees, bagging vs boosting, random forests vs gradient boosting.
why this matters
Gradient-boosted trees win most tabular problems, so 'random forest vs XGBoost' comes up in nearly every DS interview.
A decision tree splits data on the feature and threshold that most reduce impurity (Gini or entropy for classification, variance for regression). Deep trees fit training data perfectly — low bias, very high variance.
Bagging trains many deep trees on bootstrap samples and averages them; random forests also sample features at each split to de-correlate trees. This cuts variance. Boosting trains shallow trees sequentially, each fitting the errors (gradients) of the ensemble so far. This cuts bias. XGBoost, LightGBM and CatBoost are boosting implementations.
Parallel vs sequential
bagging: tree ─┐
tree ─┼─ average (parallel, reduces variance)
tree ─┘
boosting: tree → fix errors → tree → fix errors → … (sequential, reduces bias)Interview trap
Impurity-based feature importance is biased toward high-cardinality features. Use permutation importance or SHAP for explanations.
Pro tip
For tabular data, gradient boosting is usually the strongest baseline. Trees don't need feature scaling.
key takeaways
- Bagging averages deep trees to cut variance.
- Boosting adds shallow trees sequentially to cut bias.
- Trees don't need scaled features.
check your understanding
Which statement about random forests vs gradient boosting is true?