Unit 5.6 · about 4 h
165 / 324Paired bootstrap, permutation tests and multiplicity
Two models sharing a test set produce correlated scores. Unpaired uncertainty is wider than honest, and unadjusted selection is optimistic.
- implement
- measure
You can now
- Report a paired bootstrap interval for a model difference with the resampling unit matched to the comparison.
- Run a paired permutation test with the correct permutation type and state its null in words.
- Report an effect size beside every p-value and adjust for multiplicity with the search history on record.
Covers
- Paired bootstrap intervals, permutation tests and effect sizes
- Multiplicity, search history and winner's curse
Study, in this order
3 steps
- The Bootstrap and paired-sample permutation testsSciPy - Resampling and Monte Carlo methods tutorials
The notebooks linked from this page: Tutorial 3 (The Bootstrap:
scipy.stats.bootstrapwithpaired=True, BCa and percentile intervals, standard errors) and Tutorial 2b (Paired-Sample Permutation Tests:permutation_testwithpermutation_type='samples')Depth · Stay inside Tutorials 2b and 3. The independent-sample and Monte Carlo tutorials solve different comparisons.
- ISLP Chapter 5 lab: the bootstrap standard errorISLP - resampling and multiple testing
The Bootstrap (Portfolio \(\alpha\) standard error 0.0912 via
boot_SEwithB=1000and seed 0, then the linear-regression coefficient SEs) - ISLP Chapter 13 lab: FWER, FDR and the double-dipISLP - resampling and multiple testing
Family-Wise Error Rate (Bonferroni rejects only Manager One; Holm rejects Managers One and Three at FWER 0.05; the best-versus-worst paired test at \(p = 0.038\) rises to 0.186 under Tukey's method) and False Discovery Rate (Benjamini-Hochberg on all 2,000 managers)
Focus on
- Resample pairs, not models: one index stream (
paired=True) must score both models, or shared example difficulty becomes noise. - The paired-difference test is
permutation_type='samples'(sign flips within pairs).'pairings'tests association instead;'independent'shuffles labels across groups. - Pairing shrinks intervals. In the reproduce simulation a 4.8-point gap on 500 items has a paired 95% interval about 0.038 wide, against about 0.120 when each model is resampled on its own, which then covers zero.
- Every tuned, selected or re-run comparison spends error budget. Record the search history and report Holm (FWER) or BH (FDR) adjustments beside raw p-values and effect sizes.
Reproduce closed-book
default_rng(0), \(u\) uniform on 500 items, model \(A\) right when \(u < 0.70\) and model \(B\) when \(u < 0.65\), so \(B\)'s hits are a subset of \(A\)'s, the strongest pairing. Accuracies are 0.662 and 0.614; expect a paired interval near [0.030, 0.068] against near [-0.010, 0.110] when each model is resampled independently.Traps
- Bootstrapping each model's scores with its own
rngstream. Unpaired resampling destroys the pairing and inflates the interval. - Using
permutation_type='pairings'(or'independent') for a paired model comparison. That tests correlation (or cross-group exchangeability), not the mean paired difference. - Reporting the winning p-value after many seeds, splits or metrics. In the ISLP Fund lab the best-versus-worst managers, picked after looking, give \(p = 0.038\); Tukey's method over the implied 10 comparisons raises it to 0.186.
Practice
ISLP Ch05 lab: bootstrap standard errors
Reproduce \(\alpha\) SE 0.0912 (B=1000, seed 0) and the OLS bootstrap SEs from The Bootstrap
ISLP Ch13 lab: Bonferroni, Holm and Benjamini-Hochberg
Reproduce the Bonferroni and Holm verdicts (only Manager One, then One plus Three at FWER 0.05), the Tukey adjustment from 0.038 to 0.186, and the Benjamini-Hochberg rejections on all 2,000 managers
Run both labs before your own paired comparison, then report it with an effect size and a comparison count.
Self-test
-
Two models share the test set. Why must the bootstrap indices be identical for both?
Scores on the same items correlate through item difficulty. Identical indices keep the pairing, so shared difficulty cancels in the difference; independent streams add it as noise. -
Which
permutation_typetests a paired accuracy difference, and what does it shuffle?'samples': within each pair it swaps the two models' outcomes (sign flips). The null is a zero mean paired difference. -
Your paired interval is
[0.030, 0.068]and the permutation \(p\) is 0.0002 from 9999 resamples. What do you report?A 3.0-to-6.8 point gain (observed 4.8) separated from zero, the effect size, the count of comparisons that preceded it, and \(p\) below 0.001. With 9999 resamples 0.0002 is the smallest two-sided \(p\) the test can return, so it is a floor, not a measurement. -
After 20 metric looks, one comparison gives \(p = 0.03\). Ship it?
No. With 20 looks the family-wise error is near 1 minus \(0.97^{20}\) = 0.46. Adjust with Holm or BH, or pre-register the primary comparison, and report the effect size either way.
Completion criterion
Stop when
Your paired bootstrap and permutation test agree on a model comparison, the interval uses one index stream, and every p-value ships with an effect size and a comparison count.
Mark passed, then open 5.7 Only mark this when you can do it without notes.Leads to
- 5.10 Ablations, negative controls, threats to validity and decision memos Fold the paired interval into the ablation runner.
- 5.7 Learning curves, error taxonomies and slices Localize the significant gap to slices.
Comes after
- 3.6 Laws of large numbers, CLT, concentration and Monte CarloApply concentration thinking to paired resampling comparisons.
- 4.4 Bootstrap, permutation tests and multiple comparisonsApply these sampling and error-control principles to paired model comparisons.
- 5.1 Holdout, cross-validation and resampling by the right unitPut paired uncertainty on the resampled estimates.
- 5.4 Calibration curves and Platt or isotonic recalibrationPut uncertainty on the calibration comparison.
- 5.5 Cost-sensitive thresholds, imbalance and prevalence changeTest whether the thresholded gap is real or noise.
- 5.10 Ablations, negative controls, threats to validity and decision memosGive every ablation cell paired uncertainty.
Resources for this unit
- ISLP - resampling and multiple testing Use Chapter 5 and the relevant selection-history parts of Chapter 13 with their Python labs. Connect resampling to uncertainty and multiplicity without repeating the full statistics stage.
- SciPy - Resampling and Monte Carlo methods tutorials Work the notebooks linked from this page: Tutorial 3 (The Bootstrap) with
scipy.stats.bootstrap, includingpaired=True, and Tutorial 2b (Paired-Sample Permutation Tests) withpermutation_testandpermutation_type='samples'. Keep one resampling index stream across compared models.
Also in this section
- scikit-learn - model selection and evaluation route Follow Cross-validation, Hyperparameter tuning, Decision threshold tuning, Metrics and scoring, Probability calibration, Validation curves, Permutation importance and Common Pitfalls. Retain raw predictions and independently hand-check central metrics.
- Classical ML Scientific Investigation Use Core requirements, architecture constraints, Milestones A-F and failure injection as the standard for end-to-end evidence. The full project remains a later capstone route.
- D2L - Environment and Distribution Shift Read 4.7, then construct one controlled covariate, label and concept shift. Record what changes, which signal detects it and what response the evidence supports.
- Deep-ML candidate - evaluation problems After the platform admission check, use at most one of 18 K-Fold CV, 260 Expected Calibration Error, 276 ROC, 277 AUC or 840 Stratified K-Fold. Hand-checked local metrics remain the evidence.
- El-Yaniv and Wiener - On the Foundations of Noise-free Selective Classification Read the Abstract and Section 2 (Preliminary Definitions: Definitions 1 and 2 for coverage and selective risk) for the risk-coverage tradeoff. Stop before the noise-free learning bounds.