~/learn/Machine Learning · Evaluation metrics & tradeoffs
ROC, PR curves & calibration
Threshold-free evaluation, ROC-AUC vs PR-AUC on imbalanced data, and calibrated probabilities.
why this matters
'Why not just use ROC-AUC?' is a classic follow-up on fraud, ads and anomaly problems where positives are rare.
A classifier outputs a score; the threshold turns it into a label. The ROC curve plots true positive rate against false positive rate across every threshold. ROC-AUC is the probability that a random positive is scored above a random negative — 0.5 is random, 1.0 is perfect ranking.
The PR curve plots precision against recall. When positives are rare, the false positive rate stays tiny even with many false alarms, so ROC-AUC looks flattering. PR-AUC exposes that, because precision collapses as false positives pile up. A random PR-AUC baseline equals the positive rate, not 0.5.
Calibration asks whether a score of 0.8 means an 80% chance. Ranking metrics ignore it; pricing, risk and expected-value decisions need it. Fix with Platt scaling or isotonic regression and check with a reliability plot or Brier score.
Interview trap
High AUC does not mean good probabilities. Boosted trees and SVMs often rank well but are poorly calibrated.
Pro tip
For rare events (fraud, conversions), report PR-AUC plus precision at a fixed recall that matches the business operating point.
key takeaways
- ROC-AUC measures ranking quality across thresholds.
- PR-AUC is more honest when positives are rare.
- Calibration checks whether scores are real probabilities.
check your understanding
Only 0.1% of transactions are fraud. Which metric best reflects model quality?