Learning curves, error taxonomies and slices

1. Validation curves and learning curves

   scikit-learn - model selection and evaluation route

   Sections 3.5.1 (validation_curve and ValidationCurveDisplay: high train with low validation means overfit) and 3.5.2 (learning_curve and LearningCurveDisplay: Naive Bayes flat and low versus SVM converging)

   Depth · Skip the scalability example. The digits snippet numbers in Reproduce are your check.

   Done
2. Slice, error and data-size requirements

   Classical ML Scientific Investigation

   Milestones B (record predictions and errors per item), C (prescribed ablations and learning/data-size probes) and E (subgroup, time and source slices with paired uncertainty and practical significance)

   Done

Focus on

• Both scores low means bias (capacity or features); train high with validation low means variance. Validation above train is rare and usually means the two are measured differently: augmentation or dropout at train time, an easier or tiny validation split, or leakage.
• Never tune on the validation curve's peak and report that peak. The curve diagnoses; a fresh split evaluates.
• Every slice needs its support count beside the rate: 100% error on \(n=3\) is a hint, not a finding.
• Slice by deployment axes (source, time, subgroup), not by the model's own confidence alone, or the taxonomy confirms itself.

Reproduce closed-book

Traps

• Concluding that more data will fix flat converging curves. The Naive Bayes panel shows both scores stuck low: capacity, not data.
• Slicing until a bad slice appears, then reporting it raw. Unplanned slices are multiple comparisons, so pre-register slices or adjust.
• Averaging slice metrics without support weights while claiming deployment impact. Impact is rate times slice share.

Practice

Self-test

1. Train 1.00 with validation 0.82 at small \(n\), then 1.00 with 0.93 at full \(n\). Diagnosis?

   Show answer Got it Missed

2. Both curves sit flat at 0.70 from 10% to 100% of data. Next step?

   Show answer Got it Missed

3. Slice A is 2/3 wrong. Slice B is 40/400 wrong. Which finding is stronger?

   Show answer Got it Missed

4. Why slice by source or time rather than by low model confidence?

   Show answer Got it Missed

Leads to

• 5.8 Permutation importance and partial dependence Explain the worst slice with importance and dependence views.
• 5.10 Ablations, negative controls, threats to validity and decision memos Test the slice hypothesis with controls.

Comes after

• 2.5 Residual, leverage, separation and misspecification diagnosticsFold residual and slice analysis into the evaluation harness.
• 3.4 XGBoost, LightGBM and CatBoost configurationDiagnose the early-stopped runs with learning curves.
• 5.5 Cost-sensitive thresholds, imbalance and prevalence changeFind where the thresholded errors concentrate.
• 5.6 Paired bootstrap, permutation tests and multiplicityLocalize the significant gap to slices.

Resources for this unit

• scikit-learn - model selection and evaluation route
  scikit-learnprimarycourse
  Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
• Classical ML Scientific Investigation
  Roadmap-Finalreferenceassessment
  Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.