Section 5 of 5 · Classical Machine Learning

Evaluation, Uncertainty, Shift & Decisions

A score becomes actionable only after selection, uncertainty, calibration, slices, costs and plausible shifts have been checked under a frozen protocol.

10 units 20 topics 0/10

Units

10 in this section

5.1 Holdout, cross-validation and resampling by the right unit
implement
3 h
5.2 Regression error and confusion-matrix metrics
derive
3 h
5.3 ROC AUC, average precision, log loss and Brier score
derive · measure
3 h
5.4 Calibration curves and Platt or isotonic recalibration
implement · measure
4 h
5.5 Cost-sensitive thresholds, imbalance and prevalence change
derive · measure
4 h
5.6 Paired bootstrap, permutation tests and multiplicity
implement · measure
4 h
5.7 Learning curves, error taxonomies and slices
measure
3 h
5.8 Permutation importance and partial dependence
measure
3 h
5.9 Covariate, label and concept shift, drift and label delay
explain · measure
4 h
5.10 Ablations, negative controls, threats to validity and decision memos
measure
3 h

Before moving on

Checkpoint

Produce a frozen comparison with paired uncertainty, calibration or a reason it does not apply, threshold policy, slices, one shift study and a decision that survives a small movement in the best score.

Deliberately out of scope

Leave for later

Full causal and policy evaluation
Production online experimentation and monitoring
Formal conformal and fairness frameworks

Resources for this section

Shared by every unit above

scikit-learn - model selection and evaluation route
scikit-learncourse

primary

Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
ISLP - resampling and multiple testing
James, Witten, Hastie, Tibshirani and Taylortheory

required

Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.
Classical ML Scientific Investigation
Roadmap-Finalassessment

reference

Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
D2L - Environment and Distribution Shift
D2L authorstheory

support

Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
Deep-ML candidate - evaluation problems
Deep-MLpractice

optional

After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
SciPy - Resampling and Monte Carlo methods tutorials
SciPy developersreference

reference

Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with scipy.stats.bootstrap, including paired=True, and Tutorial 2b (Paired-Sample Permutation Tests) with permutation_test and permutation_type='samples'. Keep one resampling index stream across compared models.
El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification
JMLRreference

reference

Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.

Apply the section

Mini-project · Frozen comparison and decision dossier (12 to 16 h)

Take two or three candidate models through a frozen protocol on one structured dataset: raw predictions, paired uncertainty, calibration, a threshold policy, slices, one controlled shift study, and a narrow decision that survives a small movement in the best score.

Steps

Freeze the protocol before any fitting: snapshot, a group- or time-aware split, nested or clean selection, one search budget per family, and a sealed test split.
Store per-row predictions for every candidate, and hand-check the central metrics on small cases before trusting library calls.
Compare candidates with a paired bootstrap interval and a paired permutation test on the same rows. Report the effect size, the comparison count and a Holm or Benjamini-Hochberg adjustment.
Calibrate on held-out data with a reliability table, ECE checked against Deep-ML 260, and a Platt or isotonic map from a disjoint split. Set the cost-derived threshold beside TunedThresholdClassifierCV on validation, with a reject option reported as risk at coverage.
Slice errors along deployment axes with support counts and intervals, and read a learning curve to decide whether more data would help.
Run one controlled shift: move one variable or the class prior yourself, detect it with a domain classifier or prior re-estimation, and test the response against the ground truth you set.
Write the decision memo, then perturb it: replace the best score with the lower end of its paired interval or rerun with another seed, and state whether the decision holds and which tripwire would reverse it. Report the test-set numbers once.

Deliverables

The frozen protocol and the raw prediction store.
The comparison report: paired uncertainty with multiplicity, calibration, threshold policy, slices and the learning curve.
The shift study with its known ground truth.
The decision memo with its tripwire, the perturbation result and the retained negative results.

Checks

external-tests · Deep-ML 260 Expected Calibration Error
rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Frozen protocol	No choice touches the test split, and every reported number traces to stored raw predictions.	A seeded test-tuned threshold is caught by an audit before the report is written.
Paired uncertainty	A paired interval and a paired permutation test are reported on the same rows, with an effect size.	The comparison count and the multiplicity adjustment are reported, and an unpaired interval shows what pairing bought.
Calibration and threshold	A held-out reliability table and a cost-derived threshold are reported.	Recalibration fitted on disjoint data improves held-out log loss or Brier, and the reject option is reported as a (risk, coverage) pair.
Slices and shift	Slices carry support counts and intervals, and one controlled shift is detected.	The shift response is checked against the simulation's ground truth, and its failure region is stated.
Decision	The memo states the decision, evidence, uncertainty and threats.	The decision survives the perturbation and names the tripwire that would reverse it.

Feeds into: Milestone artifact 3, the decision dossier: the frozen comparison, calibration and threshold policy, slices, shift study and negative result come from here.