Cost-sensitive thresholds, imbalance and prevalence change

1. Tuning the decision threshold for class prediction

   scikit-learn - model selection and evaluation route

   Sections 3.3 (0.5 and score-0 defaults), 3.3.1.1 (TunedThresholdClassifierCV, default balanced-accuracy scoring, pos_label via make_scorer), 3.3.1.2 (internal 5-fold CV; cv='prefit' only on fresh validation data) and 3.3.1.4 (FixedThresholdClassifier with FrozenEstimator)

   Depth · Run the 3.3.1.1 snippet. The page prints 0.88 for the refit model scored on its own training data (0.887 with scikit-learn 1.9.1), against an internal cross-validated best_score_ of 0.86. The gap is in-sample optimism, so trust the 0.86.

   Done
2. Selective classifier, coverage and selective risk

   El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification

   Abstract and Section 2, Preliminary Definitions (Definitions 1 and 2: coverage and selective risk, the risk-coverage tradeoff)

   Depth · Stop before the noise-free learning bounds. The definitions plus your own confidence threshold are the whole assignment.

   Done
3. Precision and recall definitions for the prevalence arithmetic

   scikit-learn - model selection and evaluation route

   Section 3.4.4.9 for the binary precision and recall definitions; the prevalence-shift rescaling is derived in Reproduce below

   Done

Focus on

• Predict positive when \(p\) times \(C_{\text{FN}}\) exceeds \((1-p)\) times \(C_{\text{FP}}\), so \(t = C_{\text{FP}} / (C_{\text{FP}} + C_{\text{FN}})\). With miss cost 10 and false-alarm cost 1, the threshold is 1/11 = 0.091, not 0.5.
• TunedThresholdClassifierCV moves only the threshold by internal CV. When the vanilla model was fit on the same data with the same seed, predict_proba, and so the ROC and PR curves, are identical before and after.
• The reject rule is a second threshold: abstain when \(\max(p, 1-p)\) is below 1 minus reject-cost over error-cost, then report risk at the achieved coverage.
• At fixed class-conditional score distributions, a prevalence shift moves precision and F1 but not recall, FPR or ROC AUC. The cost-optimal cutoff on the old scores moves too, because the calibrated probabilities change with the base rate.

Reproduce closed-book

Traps

• Tuning the threshold on the test set, or with cv='prefit' on training data. The user guide warns explicitly that tuning data must be disjoint from training data.
• Moving the threshold to fix calibration. Thresholds spend probabilities; they cannot repair them. Recalibrate first.
• Reporting selective accuracy without coverage. Accuracy on the retained 60% says nothing beside accuracy on 100%; always pair risk with coverage.

Practice

Self-test

1. Miss cost 10, false-alarm cost 1, calibrated \(p = 0.2\). Predict positive or not, and why?

   Show answer Got it Missed

2. Why do the tuned and vanilla classifiers share ROC curves?

   Show answer Got it Missed

3. Reject cost 0.2, error cost 1. When do you abstain?

   Show answer Got it Missed

4. Prevalence halves at fixed score quality. Which metrics move?

   Show answer Got it Missed

Leads to

• 5.6 Paired bootstrap, permutation tests and multiplicity Test whether the thresholded gap is real or noise.
• 5.7 Learning curves, error taxonomies and slices Find where the thresholded errors concentrate.

Comes after

• 2.8 Probability estimates, class priors and asymmetric thresholdsExtend cost-sensitive thresholds to imbalance and prevalence change.
• 3.8 Feature importance caveats and class weighting effectsSet cost-sensitive thresholds with calibrated probabilities.
• 4.7 Outlier and novelty detection, thresholds and representationCarry the threshold policy into cost-sensitive decisions under shifted prevalence.
• 5.2 Regression error and confusion-matrix metricsAct on the matrix with cost-derived thresholds.
• 5.3 ROC AUC, average precision, log loss and Brier scoreTurn scores into decisions at a cost-derived threshold.
• 5.4 Calibration curves and Platt or isotonic recalibrationSpend calibrated probabilities on thresholds and reject rules.

Resources for this unit

• scikit-learn - model selection and evaluation route
  scikit-learnprimarycourse
  Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
• El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification
  JMLRreferencereference
  Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.