Unit 5.2 · about 3 h
161 / 324Regression error and confusion-matrix metrics
Accuracy and RMSE hide the errors that cost money. The matrix and the averaging rule decide what a single number means.
- derive
You can now
- Derive MAE, RMSE, \(R^2\) and MAPE from residuals and state which error each one punishes.
- Reconstruct precision, recall, F1 and support from a confusion matrix and choose macro, micro or weighted averaging deliberately.
- Match the regression loss to the decision cost, including quantile (pinball) and relative-error losses.
Covers
- MAE, RMSE and decision-weighted regression error
- Confusion matrix, precision, recall and F1
Study, in this order
2 steps
- Classification metrics: confusion matrix, report, precision and recallscikit-learn - model selection and evaluation route
Sections 3.4.4.6 (confusion matrix rows are true classes,
ravelordertnfpfntp,normalizemodes), 3.4.4.7 (classification report with support) and 3.4.4.9 (precision, recall, F measures and averaging rules)Depth · Skip top-\(k\) accuracy, kappa, Hamming and Jaccard unless your problem needs them.
- Regression metrics including decision-weighted lossesscikit-learn - model selection and evaluation route
Sections 3.4.6.1 (\(R^2\), including the constant-mean 0.0 and negative cases), 3.4.6.2 and 3.4.6.3 (MAE, MSE, with RMSE via
root_mean_squared_error), 3.4.6.5 (MAPE) and 3.4.6.10 (pinball loss)Depth · Skim Poisson, Gamma and Tweedie deviances. Read them fully only for count or skewed-positive targets.
Focus on
- Rows of
confusion_matrixare true classes;ravelgivestn,fp,fn,tpin that order. - \(R^2\) is 1 minus residual over total variance: 0.0 for the mean predictor, negative for worse-than-mean predictions.
- Macro averages classes equally, weighted averages by support, micro pools the counts. On imbalance they disagree, and the choice encodes costs.
- Pinball loss at quantile \(\tau\) prices over- and under-prediction asymmetrically; MAPE divides by the target, so near-zero targets dominate it.
Reproduce closed-book
y_true [0,0,0,1,1,1,1,1] with y_pred [0,1,0,1,0,1,0,1]: tn, fp, fn, tp is 2, 1, 2, 3, so precision is 0.75, recall is 0.6 and F1 is 0.667. Compute \(R^2\) on the guide's regression example, y_true [3, -0.5, 2, 7] with y_pred [2.5, 0.0, 2, 8]: 0.949.Traps
- Reading accuracy on an 80/20 split. The constant classifier scores 0.80, so report balanced accuracy or the full matrix.
- Averaging F1 with the wrong rule. Weighted averaging lets the majority class hide a failing rare class, and micro pools the counts so the rare class barely registers; macro gives it full weight. Name the rule.
- Comparing RMSE across targets with different units or scales. It carries the target unit, so normalize before comparing.
Practice
scikit-learn worked examples: confusion matrix and classification report
Reproduce the 3.4.4.6 ravel example (2, 1, 2, 3) and the 3.4.4.7 report (accuracy 0.60, macro average F1 0.49) by hand, then match with code
Do both by hand first. The macro average is the check that your averaging rule is right.
Self-test
-
tn,fp,fn,tpis 2, 1, 2, 3. What are precision, recall and F1?Precision \(3/4 = 0.75\), recall \(3/5 = 0.6\), \(\text{F1} = 2(0.75)(0.6)/1.35 = 0.667\). -
When is \(R^2\) negative, and what does that say?
When \(SS_{\text{res}}\) exceeds \(SS_{\text{tot}}\): the predictions are worse than the constant mean of the scored targets. On held-out data that can happen honestly, through overfitting or shift; in-sample, least squares with an intercept cannot go below 0. -
Why can macro-F1 and weighted-F1 rank two models differently?
Macro weights classes equally so a rare class counts fully; weighted weights by support so the majority dominates. They encode different costs. -
Your target spans cents to millions. Why is RMSE alone a poor objective?
Squared errors on the largest targets dominate it. A relative loss such as MAPE, a quantile loss, or modeling log amounts matches the decision better.
Completion criterion
Stop when
Your hand-computed matrix, report averages and \(R^2\) match sklearn on a fresh split, and you can defend the averaging rule from the class costs.
Mark passed, then open 5.3 Only mark this when you can do it without notes.Leads to
- 5.3 ROC AUC, average precision, log loss and Brier score Compare models across thresholds with ranking and proper scores.
- 5.5 Cost-sensitive thresholds, imbalance and prevalence change Act on the matrix with cost-derived thresholds.
Comes after
- 1.2 Targets, losses, metrics, decision rules and problem framingDeepen error and confusion-matrix metrics after framing them here.
- 1.7 Baselines: constant, frequency, heuristic, last-value and simple learnedCarry the fixed-harness discipline into error metrics.
Resources for this unit
- scikit-learn - model selection and evaluation route Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
Also in this section
- ISLP - resampling and multiple testing Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.
- Classical ML Scientific Investigation Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
- D2L - Environment and Distribution Shift Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
- Deep-ML candidate - evaluation problems After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
- SciPy - Resampling and Monte Carlo methods tutorials Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with
scipy.stats.bootstrap, includingpaired=True, and Tutorial 2b (Paired-Sample Permutation Tests) withpermutation_testandpermutation_type='samples'. Keep one resampling index stream across compared models. - El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.