Unit 5.4 · about 4 h
163 / 324Calibration curves and Platt or isotonic recalibration
Thresholds, costs and abstention all read probabilities. An uncalibrated 0.9 is a rumor, not a measurement.
- implement
- measure
You can now
- Read a reliability curve bin by bin and summarize it with ECE without letting one number hide the shape.
- Fit Platt (sigmoid) and isotonic recalibration on held-out predictions and explain their slope, intercept and monotonicity behavior.
- Keep a final test split untouched through fitting, calibration and threshold choice.
Covers
- Calibration curves, slope and intercept at interpretation level
- Platt and isotonic calibration on held-out predictions
Study, in this order
2 steps
- Probability calibration: curves and the calibratorscikit-learn - model selection and evaluation route
Sections 1.16.1 (reliability diagrams via
CalibrationDisplayandcalibration_curve) and 1.16.2 to 1.16.3 (CalibratedClassifierCV: ensemble averaging versus a single refit, disjoint calibration data)Depth · Skip the multiclass example unless your task has more than two classes.
- Sigmoid, isotonic and temperature scalingscikit-learn - model selection and evaluation route
Sections 1.16.3.1 (Platt sigmoid: logistic slope and intercept, preserves ranking and AUC), 1.16.3.2 (isotonic: free monotonic steps, stronger past about 1000 points, ties can change AUC) and 1.16.3.4 (temperature scaling)
Focus on
- Calibrate on predictions the classifier did not train on: CV inside
CalibratedClassifierCV, or a disjoint calibration split withFrozenEstimator. - In sklearn 1.9,
cv=Nonemeans 5-fold, stratified whenyis binary or multiclass, andensemble='auto'means an averaging ensemble of per-fold pairs unless theestimatoris aFrozenEstimator. Posts written before 1.6 showensemble=Trueas the default. - Sigmoid fits a logistic slope and intercept: safe on small samples. Isotonic fits a free step function: stronger on large calibration sets, but flat steps tie scores and can move AUC.
- Lower Brier or log loss does not prove better calibration. Both mix calibration with discrimination, so read the curve.
Reproduce closed-book
default_rng(1), \(n = 4000\), \(p\) uniform on \([0.05, 0.95]\), \(y\) drawn as \(\operatorname{Bernoulli}(p)\), score \(s = \operatorname{clip}(0.5 + 2.2(p - 0.5), 0.01, 0.99)\). Fit a one-feature logistic (Platt) map on the first 2000 rows and score the last 2000: Brier falls from 0.2130 to 0.1892 (slope 2.80, intercept -1.46 on the raw score). Bin the held-out half with sklearn.calibration.calibration_curve at 5 bins: the top bin predicts 0.964 while 0.776 are positive.Traps
- Calibrating on training predictions. Outputs pushed to 0/1 teach the calibrator the wrong map; use CV or a disjoint split.
- Judging calibration by ECE alone. One number averages away the shape; the user guide warns that lower Brier need not mean better calibrated.
- Fitting isotonic on a few hundred points and shipping it. It overfits small calibration sets, so prefer sigmoid there.
Practice
Deep-ML 260: Calculate Expected Calibration Error (ECE)
Implement expected_calibration_error and pass both public tests ([0,1] with [0.0,1.0] gives 0.0; four 0.8-probability items with two positives give 0.3)
Use equal-width bins, put \(p = 1.0\) in the last bin, skip empty bins and weight by bin share. Then apply the same function to your Platt experiment above.
Self-test
-
A bin has mean predicted probability 0.8 with 10 of 40 positives, out of 200 items total. What is its ECE contribution?
The gap |0.8 - 0.25| = 0.55 weighted by the bin share 40/200 = 0.2, so 0.11. -
Why does isotonic recalibration risk changing AUC while Platt cannot?
Isotonic is stepwise: distinct scores can land on one flat step as ties, which changes ranking. Sigmoid is strictly increasing, so pairwise order survives. -
What goes wrong if you tune the threshold on the calibration split and report on it too?
The threshold overfits that split's noise. Tune on validation or calibration data, then report once on the untouched test split. -
GaussianNBprobabilities pile near 0 and 1. Sigmoid or isotonic, and on what data?With ample held-out data, isotonic corrects the characteristic S-distortion better; with little data use sigmoid. Never fit either on the training predictions.
Completion criterion
Stop when
Your reliability table, ECE function and Platt versus isotonic comparison reproduce on held-out data, and the final test split is still sealed.
Mark passed, then open 5.5 Only mark this when you can do it without notes.Leads to
- 5.5 Cost-sensitive thresholds, imbalance and prevalence change Spend calibrated probabilities on thresholds and reject rules.
- 5.6 Paired bootstrap, permutation tests and multiplicity Put uncertainty on the calibration comparison.
Comes after
- 3.5 Bayes rule, conjugate priors and posterior predictivesCalibrate the posterior probabilities produced here before trusting them.
- 4.5 Regression diagnostics, calibration and uncertaintyApply the calibration property to held-out reliability diagrams and recalibration.
- 2.8 Probability estimates, class priors and asymmetric thresholdsRecalibrate the estimates this unit learns to distrust.
- 5.3 ROC AUC, average precision, log loss and Brier scoreRepair the probabilities that ranking scores cannot judge.
Resources for this unit
- scikit-learn - model selection and evaluation route Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
Also in this section
- ISLP - resampling and multiple testing Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.
- Classical ML Scientific Investigation Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
- D2L - Environment and Distribution Shift Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
- Deep-ML candidate - evaluation problems After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
- SciPy - Resampling and Monte Carlo methods tutorials Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with
scipy.stats.bootstrap, includingpaired=True, and Tutorial 2b (Paired-Sample Permutation Tests) withpermutation_testandpermutation_type='samples'. Keep one resampling index stream across compared models. - El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.