~/learn/Machine Learning · Evaluation metrics & tradeoffs
Regression & ranking metrics
What MAE, MSE, RMSE and R² actually mean in plain English, plus ranking metrics like hit rate, MRR and NDCG.
why this matters
Forecasting, pricing and recommendation interviews all ask how you'd measure success — and why one metric beats another.
Regression means predicting a number: a house price, a delivery time in minutes, next month's sales. To judge the model we compare two things for every row: the actual value (written y) and the model's guess (written ŷ, said 'y-hat'). The gap between them, y − ŷ, is called the error or residual.
Why we can't just average the errors
house actual (y) predicted (ŷ) error (y − ŷ) A 300k 280k +20k (guessed too low) B 250k 270k −20k (guessed too high) C 400k 390k +10k plain average of errors = (+20 − 20 + 10) / 3 ≈ +3k ← looks almost perfect!
Positive and negative errors cancel out, so a plain average hides how wrong the model really is. Every regression metric fixes this by removing the sign — either by taking the absolute value (drop the minus sign) or by squaring (a negative times a negative is positive).
MAE — Mean Absolute Error. Drop the minus signs, then average. In the example: (20 + 20 + 10) / 3 ≈ 13.3k. It reads in plain English: 'on average, our price guess is off by about $13k.' Every dollar of error counts the same, so one wild outlier can't dominate the score.
MSE — Mean Squared Error. Square each error, then average: (20² + 20² + 10²) / 3 = 300. Squaring makes big misses explode: an error of 10 costs 100, an error of 20 costs 400. Use it when large mistakes are much worse than small ones. The catch: the units are squared ('thousand-dollars squared'), which means nothing to a stakeholder.
RMSE — Root Mean Squared Error. Take the square root of MSE: √300 ≈ 17.3k. This brings the number back to normal units (dollars, minutes) while keeping the extra punishment for big misses. That's why RMSE is the most commonly reported regression metric.
MAE = mean(|y − ŷ|) average size of the miss, same units as y MSE = mean((y − ŷ)²) squared misses — big errors dominate RMSE = √MSE MSE converted back to units of y
Linear vs squared penalty
error size │ MAE penalty │ MSE penalty
───────────┼─────────────┼────────────
1 │ 1 │ 1
2 │ 2 │ 4
5 │ 5 │ 25
10 │ 10 │ 100
20 │ 20 │ 400 ← one bad miss outweighs many small onesQuick health check
RMSE is always ≥ MAE. If RMSE is much bigger than MAE, a few rows have huge errors — go inspect them (outliers, bad data, a missing feature).
A subtle but popular interview point: a model trained to minimize MAE learns to predict the median, while a model trained to minimize MSE learns to predict the mean. So if your data has extreme values (a few mansions among normal houses), MSE pulls predictions toward them and MAE stays steady.
R² — 'R-squared', the coefficient of determination. It answers: how much better is my model than a lazy baseline that always predicts the average? R² = 1 is perfect. R² = 0 means no better than always guessing the average. R² = 0.7 means the model explains 70% of the variation in y.
R² = 1 − (model's squared errors) / (squared errors of always predicting the mean) = 1 − SSE / SST
MAPE — Mean Absolute Percentage Error — expresses the miss as a percentage of the true value ('we're off by 8% on average'). Business teams love it, but it breaks when true values are zero or tiny (dividing by nearly zero) and it punishes over-predictions and under-predictions unevenly.
Ranking metrics are different: in search results, a Netflix row or a Spotify playlist, the exact score doesn't matter — the order does. What matters is whether the good items show up at the top.
Hit rate@k: did at least one relevant item appear in the top k? (k is just 'how many slots we show', e.g. top 10.) It's a simple yes/no averaged over users. MRR — Mean Reciprocal Rank: look at the position of the first relevant item and score 1 / position. First place = 1, second = 0.5, fifth = 0.2. Then average across searches. Great when users want one right answer.
NDCG@k — Normalized Discounted Cumulative Gain. Read it word by word. Gain: each item gets a relevance score (3 = perfect, 1 = so-so, 0 = useless). Discounted: items lower on the list are divided by log₂(position + 1), because fewer people scroll that far. Cumulative: add them up for the top k. Normalized: divide by the score of the perfect ordering, so the result lands between 0 and 1.
NDCG worked example
position item relevance discount contribution
1 3 ÷ log₂(2)=1.00 3.00
2 0 ÷ log₂(3)=1.58 0.00
3 2 ÷ log₂(4)=2.00 1.00
DCG = 4.00
ideal order (3, 2, 0): 3.00 + 2/1.58 = 4.26 = IDCG
NDCG = 4.00 / 4.26 ≈ 0.94Interview trap
R² can be negative on a test set — it means the model does worse than simply predicting the average. Don't report it as 'basically zero'; it's a red flag for overfitting or a broken pipeline.
Units trap
Never compare MSE or RMSE across different targets or units. An RMSE of 5 is excellent for house prices in thousands and terrible for star ratings out of 5.
key takeaways
- MAE is robust; MSE/RMSE punish big misses.
- R² compares against predicting the mean.
- NDCG and hit rate measure ranking quality.
check your understanding
A few huge errors are very costly for your business. Which loss fits best?