Bias-variance tradeoff & overfitting

Start with an everyday picture. Imagine three students preparing for an exam. The first only skims the chapter titles — they get questions wrong in practice and in the real exam (underfitting). The second memorises every past-paper answer word for word — perfect on practice papers, lost when the questions change (overfitting). The third learns the underlying ideas — good on both. A machine-learning model can behave like any of these three students.

Bias is the error from wrong assumptions: the model is too simple to capture the real pattern, like fitting a straight line to a curved relationship. Variance is the error from being too sensitive to the particular training sample: retrain on slightly different data and the predictions swing a lot. Noise is randomness in the data itself (measurement error, unpredictable behaviour) that no model can remove.

Expected error = Bias² + Variance + Irreducible noise

Why is it a tradeoff? Making a model more flexible (more features, deeper trees, more layers, higher-degree polynomials) lets it bend to the true pattern, so bias falls — but it also lets it bend to random quirks of the training rows, so variance rises. Total error is lowest somewhere in the middle.

error
  │ \                      validation
  │  \___           ______/
  │      \_____ ___/
  │  training  \______________
  └──────────────────────────── model complexity
     underfit      sweet spot     overfit

A concrete example: predicting house price from size. A straight line (degree 1) misses the fact that prices flatten for very large houses — high bias. A degree-15 polynomial passes through every training house exactly but swings wildly between them and predicts nonsense for a new house — high variance. A degree-2 or 3 curve captures the bend without chasing every point.

model            train RMSE   validation RMSE   diagnosis
degree 1 line      €48k          €50k          high bias (underfit)
degree 3 curve     €21k          €24k          good balance
degree 15 curve     €2k          €95k          high variance (overfit)

The diagnosis rule of thumb: compare training error to two reference points — the error you'd accept (a human or business baseline) and the validation error. A big gap between training error and the acceptable target means bias. A big gap between training error and validation error means variance. Both gaps can exist at once.

target error 5%

train 15%, val 16%  → bias problem      (far from target, small gap)
train  1%, val 14%  → variance problem  (close to target, big gap)
train 12%, val 25%  → both              (fix bias first, then variance)
train  5%, val  6%  → healthy — ship it, then monitor

Learning curves show the same thing across training-set size. Plot training and validation error as you train on 10%, 20% … 100% of the data. In high bias, both curves rise and fall to meet quickly and flatten at a high error — more data won't help. In high variance, a wide gap remains, but validation error is still falling as data grows — more data will help.

HIGH BIAS                       HIGH VARIANCE
error                           error
 │ val ‾‾‾‾\____________         │ val ‾‾‾\___
 │           ===========  high   │           \____  ← still falling
 │ train ___/‾‾‾‾‾‾‾‾‾‾          │                  gap
 │                               │ train ________________ low
 └──────────── training size     └──────────── training size

Fixing high bias (underfitting): add better or more informative features, use a more expressive model (trees or boosting instead of a linear model, a deeper network), reduce regularization strength, or train longer if the model hasn't converged.

Fixing high variance (overfitting): collect more training data, apply regularization (L1/L2 penalties, dropout, weight decay), simplify the model (shallower trees, fewer features, fewer parameters), use early stopping, use bagging/ensembles such as random forests that average away variance, and add data augmentation for images or text.

symptom              first moves
high bias            + features, + capacity, − regularization
high variance        + data, + regularization, − capacity,
                     early stopping, bagging, augmentation
both                 fix bias first, then control variance

How common algorithms map to the tradeoff: linear/logistic regression are low variance but can be high bias. A single deep decision tree is low bias but high variance. Random forests reduce variance by averaging many trees. Gradient boosting reduces bias by adding trees that correct earlier mistakes — which is why it can overfit if you add too many or use a large learning rate. In k-nearest neighbours, small k = high variance, large k = high bias.

Hyperparameters are your tradeoff dials. Regularization strength λ: higher → simpler model, less variance, more bias. Tree max_depth: deeper → less bias, more variance. Number of boosting rounds: more → less bias, more variance (control with early stopping). Dropout rate: higher → less variance. Tune these on a validation set or with cross-validation, never on the test set.

Interview-ready answer to 'Your model is overfitting — what do you do?': (1) confirm it — compare train vs validation error and check for leakage; (2) plot learning curves to see whether more data would help; (3) try regularization, a simpler model, or early stopping; (4) get more or augmented data if curves suggest it; (5) re-evaluate on a held-out test set and monitor in production, because drift can make a good model overfit to the past.