~/learn/Machine Learning · Foundations & generalization

Validation strategies

Train/validation/test splits, k-fold cross-validation, time-based splits and data leakage.

5 min readlesson 3 of 15cross-validationleakagetime split

why this matters

Leakage is the number-one reason models that look great offline fail in production — interviewers love to probe for it.

You measure generalization on data the model never saw. Train fits parameters, validation picks hyperparameters, and the test set is touched once at the end. K-fold cross-validation rotates the validation fold to get a more stable estimate on small data; stratified k-fold keeps class ratios equal per fold.

Leakage is when information unavailable at prediction time sneaks into training: fitting a scaler on the full dataset, features computed after the label event, or the same user appearing in train and test. Leakage produces great offline scores that collapse in production.

Split like production

random split   ── fine for i.i.d. rows
group split    ── same user/patient never in both sides
time split     ── train on past, validate on future
                  [ train ......... ][ val ][ test ]  → time

Interview trap

Never randomly shuffle time series. A model trained on next month's rows to predict last month's looks brilliant and is useless.

Pro tip

Fit every preprocessing step (scalers, imputers, encoders) inside the CV loop or a pipeline, on the training fold only.

key takeaways

Split the way production will see data.
Use time splits for temporal data, group splits for repeat entities.
Fit preprocessing on training folds only.

check your understanding

You predict next week's sales. Which validation scheme is right?

Bias-variance tradeoff & overfitting Classification metrics