Split design, nested selection and test isolation

1. Test-set construction: random, hashed, and stratified splits

   Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2

   Book Chapter 2: Create a Test Set (printed pp. 57-62). Notebook 02: Create a Test Set: shuffle_and_split_data with seed 42, crc32 id-hash split with the longitude/latitude stable-id pattern, train_test_split with random_state 42, the 10.7 percent bad-sample binomial calculation with its simulation check, income_cat binning with pd.cut, StratifiedShuffleSplit, the single stratified train_test_split call, and the Overall/Stratified/Random proportion table

   Depth · Use the housing income stratification as the method demo. Your graded split is the chronological Bike rebuild, not housing.

   Done
2. Selection inside validation, test set touched once

   Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2

   Book Chapter 2: Fine-Tune Your Model through Evaluate Your System on the Test Set (printed pp. 94-100). Notebook 02: Grid Search and Randomized Search through Evaluate Your System on the Test Set: cross-validated search, best-model error analysis, final test RMSE with the scipy bootstrap 95 percent confidence interval; plus the Chapter 1 Exercise Solutions 16 to 19 on test, validation, and train-dev roles and why tuning on the test set overfits it

   Depth · Read the selection-then-final-test order. Do not run the full search here.

   Done
3. Group-aware, time-aware, and single train/test splits

   scikit-learn - Cross-validation splitters

   3.1.2.4 Cross-validation iterators for grouped data with 3.1.2.4.1 Group K-fold (groups [1, 1, 1, 2, 2, 2, 3, 3, 3, 3] giving test folds [6, 7, 8, 9], [3, 4, 5], [0, 1, 2]); 3.1.2.5 Using cross-validation iterators to split train and test (GroupShuffleSplit with random_state 7); 3.1.2.6 Cross validation of time series data with 3.1.2.6.1 Time Series Split (n_splits 3 on 6 samples); 3.1.3 A note on shuffling

   Done

Focus on

• Choose the splitter from the dependence structure. Stratify when a rare subgroup must survive (income_cat); group when rows share an entity (GroupKFold holds each of groups 1, 2, 3 out in turn); go temporal when deployment runs forward in time (TimeSeriesSplit trains [0, 1, 2] tests [3], then expands).
• Stratification needs discrete labels, so regression targets get binned first: pd.cut on median_income into 5 cats, stratify, then drop the bin column before training. The Overall/Stratified/Random proportion table is the oracle.
• Selection lives inside validation and the test set is touched once. The HOML order is cross-validated search, error analysis, then one final test RMSE with a bootstrap interval; the checklist rule is never tweak after measuring generalization error.
• A random split of bike hours leaks the future: neighboring hours share weather and demand. The honest deployment proxy is the chronological cut at 2012-01-01, giving 8645 train rows and 8734 test rows.

Reproduce closed-book

Traps

• Shuffling time-ordered rows before splitting. Section 3.1.3 warns that shuffling news-like data overfits near-time similarity; bike hours need the chronological cut, not a shuffle.
• Stratifying on the raw regression target. stratify needs classes, so bin first with income_cat and drop the bin column before training, exactly as the notebook cells do.
• Reusing the test set for selection and reporting it as generalization. Every peek costs the guarantee; fix split identities from the start and let lineage make that mechanical.

Practice

The section mini-project covers this: the chronological Bike split (2011 train, 2012 test) with assertions that no future timestamp and no prohibited group cross the boundary.

Self-test

1. Housing data holds few high-income districts. Why does the notebook bin median_income before splitting?

   Show answer Got it Missed

2. What is the 10.7 percent number in Create a Test Set?

   Show answer Got it Missed

3. Bike rows from the same week share weather. Which split answers generalization to next year?

   Show answer Got it Missed

4. Grid search has picked hyperparameters on validation folds. What may you now do with the test set?

   Show answer Got it Missed

Leads to

• 1.4 Leakage: target, future, entity, duplicate, preprocessing and feedback Name every way the split boundary can still leak after it is drawn.
• 1.6 Training-only imputation, scaling, encoding and learned state Fit every learned preprocessing state inside the split built here.

Comes after

• 1.1 Unit of observation, prediction time and label horizonDerive the split from the time and entity boundaries fixed here.
• 1.2 Targets, losses, metrics, decision rules and problem framingEvaluate the chosen metric under a split that respects the decision process.
• 5.1 Holdout, cross-validation and resampling by the right unitMatch the split design to the same dependence structure.

Resources for this unit

• Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2
  Aurelien Geron (O'Reilly, 2025)primarycourse
  Read Chapters 1 and 2 of the book, then run notebooks 01 and 02 and use ml-project-checklist.md for framing, data inspection, test-set creation, preprocessing pipelines, cross-validation, error analysis and persistence. Rebuild the workflow on UCI Bike Sharing with a chronological split and prohibit casual and registered when predicting cnt.
• scikit-learn - Cross-validation splitters
  scikit-learnsupportreference
  Use the group-aware and time-aware splitter sections plus GroupKFold and TimeSeriesSplit. Choose from the dependence structure and assert that no prohibited group or future timestamp crosses the boundary.