Ablations, negative controls, threats to validity and decision memos

1. Ablations, failure injection and defense

   Classical ML Scientific Investigation

   Milestones C (prescribed ablations under one controlled search budget), D (inject target-proxy, duplicate, future-feature, global-preprocessing and test-tuned-threshold failures; audits must fail at the right boundary; preserve failed runs), E (effects with paired uncertainty and multiplicity) and the Design-defense questions (baseline, metric-threshold match, what each ablation isolated, the largest remaining shift)

   Done
2. Dummy estimators as null controls

   scikit-learn - model selection and evaluation route

   Section 3.4.8 (most_frequent, stratified and uniform baselines that ignore the features; the worked example sets the bar the real model must clear)

   Done
3. Common pitfalls: leakage patterns that audits must catch

   scikit-learn - model selection and evaluation route

   Section 12.2 (split-first discipline, scaler-in-pipeline, the SelectKBest-on-all-data pattern scoring 0.76 on random labels versus 0.50 split-first)

   Done

Focus on

• Change one mechanism per cell with data, splits and evaluation frozen. Record every variant and failure, not just the winner.
• Every claim needs a null that must fail: a dummy baseline, a shuffled-target run, or a seeded leak the audit must catch.
• Tune all compared families under one controlled search budget. Unequal tuning is a confound, not a result.
• Write the memo narrow: decision, evidence, uncertainty, threats, and what would change the call. A conclusion that breaks on a 0.005 score move is not a conclusion.

Reproduce closed-book

Traps

• Ablating by removing a component and retuning everything else freely. Retuning confounds the comparison; freeze the budget and the protocol.
• Keeping only the winning run. Without the failed variants and the search history, the winner's curse makes the margin a lie.
• A negative control that passes. If the dummy or shuffled-target run looks good, the evaluation is broken, not the model brilliant.

Practice

Self-test

1. Removing feature set X drops accuracy 0.02 with a paired interval of [-0.01, 0.05]. What do you conclude?

   Show answer Got it Missed

2. On a balanced binary task, your shuffled-target control scores 0.76. What is broken?

   Show answer Got it Missed

3. Model A had 3x the tuning budget of model B and wins by 0.01. Decision?

   Show answer Got it Missed

4. What makes a decision memo narrow?

   Show answer Got it Missed

Leads to

• 5.6 Paired bootstrap, permutation tests and multiplicity Give every ablation cell paired uncertainty.
• 1.7 Baselines: constant, frequency, heuristic, last-value and simple learned Return stronger baselines as standing controls.

Comes after

• 1.8 Snapshots, lineage, pipeline serialization and serving parityUse frozen lineage and parity as the base for ablations and controls.
• 5.6 Paired bootstrap, permutation tests and multiplicityFold the paired interval into the ablation runner.
• 5.7 Learning curves, error taxonomies and slicesTest the slice hypothesis with controls.
• 5.8 Permutation importance and partial dependenceTest an important feature by removing it under controls.
• 5.9 Covariate, label and concept shift, drift and label delayProve the shift response with ablations and controls.

Resources for this unit

• scikit-learn - model selection and evaluation route
  scikit-learnprimarycourse
  Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
• Classical ML Scientific Investigation
  Roadmap-Finalreferenceassessment
  Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.