~/learn/Machine Learning · Classical algorithms
Linear & logistic models
Least squares, log-odds, regularization with L1 (Lasso) vs L2 (Ridge).
why this matters
Linear and logistic regression are the default baselines — expect questions on coefficients, log-odds and regularization.
Linear regression predicts y = wᵀx + b by minimizing squared error. Logistic regression passes the same linear score through a sigmoid to get a probability, and is trained with log-loss. Each coefficient is the change in log-odds for a one-unit change in that feature, holding others fixed.
sigmoid(z) = 1 / (1 + e^(−z)) log-loss = −[y·log(p) + (1 − y)·log(1 − p)] Ridge (L2): loss + λ·Σ w² Lasso (L1): loss + λ·Σ |w|
Why L1 is sparse
L2 constraint = circle → loss contours touch it anywhere → small weights L1 constraint = diamond → contours hit the corners → exact zeros
L1 drives some weights exactly to zero, acting as feature selection. L2 shrinks all weights smoothly and handles correlated features more stably. Elastic Net mixes both.
Interview trap
Coefficients are only comparable when features are on the same scale, and collinear features make individual coefficients unstable even when predictions are fine.
key takeaways
- Logistic = linear score through a sigmoid, trained with log-loss.
- L1 creates sparse weights; L2 shrinks smoothly.
- Scale features before comparing coefficients.
check your understanding
You want a model that automatically drops useless features. Which regularizer?