Unit 5.3 · about 3 h
162 / 324ROC AUC, average precision, log loss and Brier score
Ranking scores compare models across thresholds, but only proper scores judge the probabilities you will later threshold, weight or abstain on.
- derive
- measure
You can now
- Derive ROC AUC as the pairwise ranking probability and average precision as the precision-recall summary.
- Explain why ROC AUC is prevalence-independent while average precision moves with the base rate.
- Compute log loss and Brier score by hand and state what each one punishes beyond ranking.
Covers
- ROC AUC, average precision, log loss and Brier score
Study, in this order
1 steps
- ROC, precision-recall, log loss and Brier scorescikit-learn - model selection and evaluation route
Sections 3.4.4.15.1 (binary ROC with
predict_proba[:, 1]ordecision_function; breast-cancer AUC 0.99), 3.4.4.9 (average_precision_scoreandprecision_recall_curve), 3.4.4.12 (log loss) and 3.4.4.18 (Brier score, binary range and calibration note)Depth · Skip multilabel ranking metrics (3.4.5) and DET curves unless your task needs them.
Focus on
- Pass the positive-class column
predict_proba(X)[:, 1](classes_[1]) ordecision_functiontoroc_auc_score, never hard predictions. - The docs' 0.99 breast-cancer AUC is a demonstration, not a result: the snippet re-scores its own training data.
- Average precision summarizes the PR curve and falls with prevalence; ROC AUC conditions within each class and does not move.
- Log loss punishes one confident error without bound; Brier stays bounded and mixes calibration with discrimination.
Reproduce closed-book
y [0,0,1,1] with scores [0.1,0.4,0.35,0.8] to get 0.75 (3 of 4 positive-negative pairs ordered), plus log loss 0.47 and Brier 0.16, and list the ROC points (0,0), (0,0.5), (0.5,0.5), (0.5,1), (1,1).Traps
- Feeding
predict()labels intoroc_auc_score. Hard labels give one elbow and a collapsed AUC; passscores. - Quoting the in-sample 0.99 AUC from the docs snippet as model quality. It re-scores training data.
- Treating ROC AUC as a calibration check. A model can rank perfectly while its probabilities are badly miscalibrated.
Practice
Hand-check ROC, AUC and Brier against sklearn
Match your hand computation on y [0,0,1,1] with scores [0.1,0.4,0.35,0.8] (AUC 0.75, AP 0.833) against roc_auc_score, average_precision_score and brier_score_loss, then reproduce the confusion_matrix_at_thresholds table (tns [2,1,1,0], tps [1,1,2,2])
All three numbers were verified with sklearn 1.9. Recompute AP by hand from the PR points before checking.
Self-test
-
What does AUC 0.75 mean as a pairwise statement on the reproduce example?
A random positive outranks a random negative with probability 0.75. Three of the four pairs are correctly ordered; only 0.35 versus 0.4 is flipped. -
Prevalence drops 10x at fixed score distributions. What happens to ROC AUC and to average precision?
ROC AUC is unchanged because it conditions within each class. Precision at fixed recall falls, so average precision drops. -
Why does log loss explode on one confident error while Brier stays bounded?
Log loss takes minus \(\log\) of the true-class probability, unbounded as it goes to 0. Binary Brier is the squared error of the positive-class probability, at most 1 per item (the guide's halved form of the two-class sum). -
roc_curvereturns thresholds starting atinf. Why isinfnot an operating threshold?The first point predicts nothing positive (FPR and TPR both 0). Inf sits above every score; the usable thresholds are the distinct scores.
Completion criterion
Stop when
Your hand AUC, AP, log loss and Brier match sklearn to 3 decimals, and you can say which metric moves when prevalence shifts.
Mark passed, then open 5.4 Only mark this when you can do it without notes.Leads to
- 5.4 Calibration curves and Platt or isotonic recalibration Repair the probabilities that ranking scores cannot judge.
- 5.5 Cost-sensitive thresholds, imbalance and prevalence change Turn scores into decisions at a cost-derived threshold.
Comes after
- 2.7 Naive Bayes: event models, smoothing and log-space scoringScore the outputs with log loss and Brier rather than accuracy.
- 5.2 Regression error and confusion-matrix metricsCompare models across thresholds with ranking and proper scores.
Resources for this unit
- scikit-learn - model selection and evaluation route Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
Also in this section
- ISLP - resampling and multiple testing Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.
- Classical ML Scientific Investigation Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
- D2L - Environment and Distribution Shift Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
- Deep-ML candidate - evaluation problems After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
- SciPy - Resampling and Monte Carlo methods tutorials Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with
scipy.stats.bootstrap, includingpaired=True, and Tutorial 2b (Paired-Sample Permutation Tests) withpermutation_testandpermutation_type='samples'. Keep one resampling index stream across compared models. - El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.