Holdout, cross-validation and resampling by the right unit

A validation score that guided model choice is optimistic. The sampling unit decides what the error bars mean at deployment.

implement

You can now

Choose \(K\)-fold, stratified, group or time-series resampling from the deployment dependence structure and name the sampling unit.
Run nested selection so the reported score never guided the choice it evaluates.
Keep preprocessing inside the resampling loop so no test information leaks into fitting.

Covers

Holdout, cross-validation, nested selection and final test isolation
Group and temporal resampling with the correct sampling unit

Study, in this order

3 steps

Cross-validation: computing metrics and choosing iterators
scikit-learn - model selection and evaluation route

Sections 3.1.1 (cross_validate versus cross_val_score, the Data transformation with held-out data note, 3.1.1.2 cross_val_predict and its warning) and 3.1.2 (K-fold, Stratified K-fold, Group K-fold, TimeSeriesSplit)

Done
Nested versus non-nested cross-validation on iris
scikit-learn - model selection and evaluation route

Whole example: inner GridSearchCV over \(C\) and \(\gamma\), outer cross_val_score; the non-nested best_score_ is optimistic next to the nested mean

Done
ISLP Chapter 5 lab: validation-set, LOOCV and k-fold errors
ISLP - resampling and multiple testing

The Validation Set Approach (linear/quadratic/cubic MSE triple 23.62, 18.76, 18.80 on Auto) and Cross-Validation (LOOCV MSE 24.23, then the 10-fold KFold comparison)

Depth · Skim the bootstrap half. It belongs to the uncertainty unit.

Done

Focus on

An integer cv in cross_val_score means StratifiedKFold when the estimator is a classifier and y is binary or multiclass, and plain KFold otherwise, unshuffled in both cases.
cross_val_predict returns one out-of-fold prediction per row for inspection and stacking, but its pooled score is not a generalization estimate.
GroupKFold keeps whole groups out of training together; TimeSeriesSplit only ever tests on the future. The deployment unit picks the splitter.
Every fitted transform (scaling, imputation, selection) must sit inside a Pipeline under CV, or test statistics shape the training features.

Reproduce closed-book

Closed-book, write the nested loop for an RBF SVC over \(C \in \{1, 10, 100\}\) and \(\gamma \in \{0.01, 0.1\}\) on iris, with KFold(4, shuffle=True) inner (random_state=0) and outer (random_state=1) splits, and state which number is honest. Reference run: nested mean 0.9468 against non-nested best_score_ 0.9603. One trial is noisy: the docs example averages 30 trials and finds a mean gap of only 0.0076 (std 0.0078).

Traps

Reporting GridSearchCV.best_score_ as model performance. It was maximized over the same folds, so the nested mean is the honest estimate.
Splitting rows when the deployment unit is a group or a time block. Shared patients, users or days across train and test inflate the score.
Fitting the scaler or selector on all data before cross_val_score. Wrap it in a Pipeline so each fold fits transforms on its train split only.

Practice

ISLP Ch05 lab: validation-set and cross-validation errors

Reproduce the linear/quadratic/cubic validation MSE triple and the LOOCV estimate 24.23 on Auto, then the 10-fold comparison

Match the printed numbers before trusting your own splitter on grouped or temporal data.

Self-test

When does an integer cv in cross_val_score stratify?
You select \(C\) by CV and report the best CV score. What is wrong?
Each patient contributes several rows. Why is plain KFold wrong, and what replaces it?
Why must scaling sit inside the CV loop?

Completion criterion

Stop when

You reproduce the nested versus non-nested gap on iris, and you can name the sampling unit and the nested structure for a grouped or temporal problem without notes.

Mark passed, then open 5.2 Only mark this when you can do it without notes.

Leads to

5.6 Paired bootstrap, permutation tests and multiplicity Put paired uncertainty on the resampled estimates.
1.3 Split design, nested selection and test isolation Match the split design to the same dependence structure.

Comes after

4.1 Estimands, estimators, bias, variance and consistencyApply sampling-distribution reasoning when rows are not independent units.
4.6 Missingness and selection mechanismsCheck population selection before treating observed units as representative.
4.7 Randomized experiment design, power and MDERespect dependence and randomization units in uncertainty estimates.
4.8 Causal claims, causal graphs, adjustment and identificationKeep identification separate from sampling uncertainty and population selection.
4.8 t-SNE and UMAP as guarded exploratory viewsMove from exploratory views to evaluation, starting with the sampling unit behind every estimate.

Resources for this unit

scikit-learn - model selection and evaluation route
scikit-learnprimarycourse
Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
ISLP - resampling and multiple testing
James, Witten, Hastie, Tibshirani and Taylorrequiredtheory
Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.

Also in this section

Classical ML Scientific Investigation
Roadmap-Finalreferenceassessment
Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
D2L - Environment and Distribution Shift
D2L authorssupporttheory
Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
Deep-ML candidate - evaluation problems
Deep-MLoptionalpractice
After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
SciPy - Resampling and Monte Carlo methods tutorials
SciPy developersreferencereference
Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with scipy.stats.bootstrap, including paired=True, and Tutorial 2b (Paired-Sample Permutation Tests) with permutation_test and permutation_type='samples'. Keep one resampling index stream across compared models.
El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification
JMLRreferencereference
Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.