Unit 5.9 · about 4 h
168 / 324Covariate, label and concept shift, drift and label delay
Test error assumes tomorrow looks like yesterday. Each shift type breaks a different link, so each needs its own detector and response.
- explain
- measure
You can now
- Classify a change as covariate, label or concept shift by naming which distribution moved.
- Build a domain-classifier covariate-shift detector and apply importance weights with the support condition checked.
- Separate data drift from performance drift and design monitoring that survives label delay.
Covers
- Covariate, label and concept shift
- Data drift, performance drift and label delay
Study, in this order
3 steps
- D2L Section 4.7: shift types and correctionsD2L - Environment and Distribution Shift
Types (covariate: \(P(y \mid x)\) fixed with \(p(x)\) moved; label: \(P(y)\) moved with \(P(x \mid y)\) fixed; concept: \(P(y \mid x)\) moved), Correction (importance weights \(\beta = p(x)/q(x)\) via a domain classifier, the label-shift confusion-matrix solve) and Taxonomy (batch versus online deployment)
Depth · Skim the anecdotes and fairness sections. Work the math in Correction.
- D2L 4.7 Exercises: detector and correctorD2L - Environment and Distribution Shift
Exercises 2 and 3: implement a covariate shift detector (hint: build a classifier) and a covariate shift corrector
- Shift probes, label delay and monitoring designClassical ML Scientific Investigation
Milestones C (shift probes) and E (slices with paired uncertainty), failure injection (label delay, shifted prevalence, group overlap) and the monitoring-design stretch goal
Focus on
- Name the moving piece: covariate moves \(p(x)\), label moves \(p(y)\), concept moves \(P(y \mid x)\). The detector and the fix follow the name.
- Covariate correction reweights by \(\beta = p_{\text{test}}(x)/p_{\text{train}}(x)\), estimated as \(\exp(h(x))\) from a train-versus-test logistic domain classifier on equal sample counts (otherwise scale by the count ratio), capped as \(\min(\exp(h), c)\). It needs train support everywhere test reaches.
- Label shift solves mean-test-predictions equals confusion-matrix times new priors. It needs \(P(x \mid y)\) fixed and a validated confusion matrix.
- Data drift without performance drift is a warning, not a failure. With label delay, performance drift stays unobservable until labels land, so monitor input drift and slice volumes meanwhile.
Reproduce closed-book
Traps
- Correcting with weights where test reaches and train never went. Importance weights cannot invent unseen regions; collect data instead.
- Reading a domain-classifier AUC near 0.5 as no shift. It may be a weak detector, so check slice volumes and the performance series too.
- Waiting for late labels to detect breakage. Design input-drift and volume monitors that fire before labels land.
Practice
D2L 4.7 Exercises: shift detector and corrector
Exercises 2 and 3: a domain-classifier covariate-shift detector and an importance-weighted corrector on a controlled synthetic shift
Control the shift yourself (one variable moves) so you know the ground truth the detector must find.
Self-test
-
Cat photos train, cartoon cats test. Which shift, and what stays fixed?
Covariate shift: \(P(y \mid x)\), what makes a cat a cat, is fixed while \(p(x)\) moved. Reweight or adapt features; the labeling rule stands. -
Disease prevalence doubles with identical symptoms per disease. Which shift, and what is the fix?
Label shift: \(P(y)\) moved with \(P(x \mid y)\) fixed. Estimate the new priors with the confusion-matrix solve and reweight by the prior ratio. -
Spammers keep crafting messages unlike anything seen before, month after month. Which kind of shift, and what helps?
Nonstationary, adversarial drift, D2L's spam example. \(p(x)\) moves into regions training never covered, and where spam imitates normal mail \(P(y \mid x)\) moves too, so it is not a clean covariate shift. Reweighting cannot help without support; relabel recent data and retrain on a schedule. -
Input drift alarms fire but held-out accuracy holds. Ship or stop?
Neither yet: data drift without performance drift is a warning. Check slice volumes, label-delay horizons and support coverage before changing anything.
Completion criterion
Stop when
You classify a new change into covariate, label or concept shift on sight, and your detector plus weighted estimate respond correctly to a synthetic shift you control.
Mark passed, then open 5.10 Only mark this when you can do it without notes.Leads to
- 5.10 Ablations, negative controls, threats to validity and decision memos Prove the shift response with ablations and controls.
- 1.5 Schema, provenance, missingness, duplicate and availability audits Feed drift monitors with audited schema and provenance.
Comes after
- 4.7 Outlier and novelty detection, thresholds and representationReuse novelty scores as drift signals with detection-delay notes.
- 5.8 Permutation importance and partial dependenceCorrelated structure is what shift breaks first.
Resources for this unit
- Classical ML Scientific Investigation Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
- D2L - Environment and Distribution Shift Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
Also in this section
- scikit-learn - model selection and evaluation route Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
- ISLP - resampling and multiple testing Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.
- Deep-ML candidate - evaluation problems After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
- SciPy - Resampling and Monte Carlo methods tutorials Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with
scipy.stats.bootstrap, includingpaired=True, and Tutorial 2b (Paired-Sample Permutation Tests) withpermutation_testandpermutation_type='samples'. Keep one resampling index stream across compared models. - El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.