Unit 1.4 · about 4 h
131 / 324Leakage: target, future, entity, duplicate, preprocessing and feedback
Leakage is the usual cause of scores that are too good to deploy. Each kind has a different audit, so taxonomy first, then tests.
- explain
- measure
You can now
- Explain target leakage, future leakage, entity overlap, duplicate contamination, preprocessing leakage, label delay, and feedback leakage.
- Diagnose an unexpectedly high score by auditing duplicate entities, time, preprocessing, label construction, selection, and metric code.
- Quantify the effect of repairing one leakage defect while keeping the model and evaluation fixed.
Covers
- Target, future, entity, duplicate and preprocessing leakage
- Label delay, selection bias and feedback leakage
Study, in this order
4 steps
- Leakage definition, repairs, and the `SelectKBest` demoscikit-learn - Common pitfalls and pipelines
12.2 Data leakage with the never-call-fit-on-test rule; 12.2.1 How to avoid data leakage (split first,
fiton train,transformboth); 12.2.2 Data leakage during pre-processing with theSelectKBestwrong run at 0.76 versus the right run at 0.50 on random labels and the pipeline cross-validation at 0.43 +/- 0.05 - Target construction in the Bike dictionaryUCI Bike Sharing Dataset
Additional Variable Information:
casual,registered, andcntdefinitions, withcntthe count of total rentals including bothcasualandregisteredDepth · Read the definitions, then verify the identity on the file yourself:
casualplusregisteredequalscnton everyhour.csvrow (verified max difference 0 on 17379 rows). - Freshness, export checks, and monitoringRules of Machine Learning - first controls
Rule 8: Know the freshness requirements of your system; Rule 9: Detect problems before exporting models, including the held-out AUC sanity check
- Sampling discipline and production monitoringHands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2
Get the data, item 11 (test set aside, no snooping); Launch: monitoring code, slow degradation and rot, input-quality checks, retraining on fresh data, and the human pipeline for labels
Focus on
- Leakage is any use of information unavailable at prediction time (section 12.2). The repair never varies: split first,
fiton train only,transformboth, preferably inside aPipeline. - The
SelectKBestdemo quantifies it. Selecting on all of \(200 \times 10000\) random features then splitting scores 0.76 against chance near 0.50; selecting on train only scores 0.50 with pipeline cross-validation at 0.43 +/- 0.05. Distance above chance on random labels is the alarm. - Bike
cntis defined ascasualplusregisteredon every row, so both are target parts, never features. A linear model on the two scores RMSE 0.0 with coefficients[1, 1]and intercept 0; the label-free 12-feature linear baseline on the same chronological split scores 192.3. - Delay, selection, and feedback are leakage over time and policy: late labels need freshness limits and a human labeling pipeline, selected training rows need the sampling discipline of unit 3, and predictions that reshape future labels need input monitoring and pre-export checks. Production feedback control stays excluded per
leaveForLater.
Reproduce closed-book
SelectKBest pair (wrong 0.76, right 0.50 on random labels) and the bike identity check (casual plus registered equals cnt on all 17379 rows, leaking linear model RMSE 0.0 with coefficients [1, 1]). State both repairs: fit selectors and scalers on train only inside a Pipeline, and drop casual and registered from cnt features.Traps
- Judging leakage by metric movement alone. Fitting the bike weather scaler on all rows versus train only gives RMSE 200.95 both ways with
LinearRegression, because least-squares predictions are unchanged by any per-column rescaling; evenRidgeand 10-nearest-neighbors move by under 0.05. The procedure is still broken. Test the procedure, not just the score. - Calling deduplication a leakage fix by itself. Removing exact duplicates without fixing split ownership still lets entity twins straddle train and test; assert group and entity disjointness after dedup, per unit 3.
- Treating feedback as a modeling bug. When serving decisions select tomorrow labels, no refit breaks the loop; the contract must log exposure and keep an unaffected holdout slice, which the mini-project records as a monitoring exclusion.
Practice
The section mini-project covers this: seeded global-scaling, target-derived-feature, and partition-duplicate defects, each caught by a failing test with the conclusion change reported after repair.
Self-test
-
SelectKBeston all \(200 \times 10000\) random features then split scores 0.76. Why is that proof of leakage rather than signal?\(X\) and \(y\) are independent by construction, so chance sits near 0.50: the clean run scores 0.50 and pipeline cross-validation gives 0.43 +/- 0.05. Only the selector seeing test labels explains 0.76. -
Predicting bike
cntfromcasualandregisteredgives RMSE 0.0. Good model?No.cntis defined as their sum on every row, so the model recites label construction with coefficients[1, 1]. Drop both; the label-free 12-feature linear baseline scores 192.3. -
Scaling bike weather on all rows versus train only leaves a
LinearRegressionRMSE at 200.95 both ways. Leak or not?Still a leak in procedure: test rows moved thetempmean from 0.4891 to 0.4970. The score cannot move here, because least-squares predictions are unchanged by any per-column rescaling, so this model is blind to the defect. Repair it anyway; with other models and splits the same habit costs more, as 0.76 versus 0.50 shows. -
Labels arrive two weeks late and serving prioritizes flagged users. Name the two risks and one guard each.
Label delay: evaluation lags reality, so set freshness limits and a human labeling pipeline (checklist Launch, Rule 8). Selection and feedback: training overrepresents flagged users and predictions reshape future labels, so log exposure, keep an unaffected holdout slice, and gate exports with Rule 9 checks. -
Copying 500 test rows into bike train moves a weather-only decision tree from RMSE 219.4 to 214.1. Small move, so harmless?
No. On the 500 copied rows the error falls from 215.3 to 113.7, while the other 8234 test rows barely move (219.6 to 218.7). The copies are under 6 percent of the test set, so the average hides them. Ban cross-partition duplicates with ownership assertions, and score contaminated rows separately when you audit.
Completion criterion
Stop when
You reproduce both number pairs (0.76/0.50 and 0.0/192.3) and can name the repair for each of the six leakage kinds without notes.
Mark passed, then open 1.5 Only mark this when you can do it without notes.Leads to
- 1.5 Schema, provenance, missingness, duplicate and availability audits Build the audit suite that catches these defects routinely.
- 1.6 Training-only imputation, scaling, encoding and learned state Put the preprocessing repair inside a pipeline.
Comes after
- 1.3 Split design, nested selection and test isolationName every way the split boundary can still leak after it is drawn.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2 Read Chapters 1 and 2 of the book, then run notebooks 01 and 02 and use ml-project-checklist.md for framing, data inspection, test-set creation, preprocessing pipelines, cross-validation, error analysis and persistence. Rebuild the workflow on UCI Bike Sharing with a chronological split and prohibit casual and registered when predicting cnt.
- scikit-learn - Common pitfalls and pipelines Read Common Pitfalls sections on inconsistent preprocessing and leakage, then Pipeline and ColumnTransformer. Seed global scaling, a target-derived feature and a partition duplicate, and make tests reject each defect.
- Rules of Machine Learning - first controls Read Rules 1-5 and 8-9. Turn the existing heuristic into a baseline and test the feature path independently of the learner.
- UCI Bike Sharing Dataset Use hour.csv as the compact integrated dataset. Preserve a raw snapshot and data dictionary, split chronologically, and document why casual and registered are target proxies for cnt.
Also in this section
- scikit-learn - Cross-validation splitters Use the group-aware and time-aware splitter sections plus GroupKFold and TimeSeriesSplit. Choose from the dependence structure and assert that no prohibited group or future timestamp crosses the boundary.
- Google ML Problem Framing - framing an ML problem Read Identify the output you need through Proxy labels and Define the success metrics, including the video-caching threshold analysis, the proxy-issue table, and both Check Your Understanding boxes.