Unit 3.8 · about 3 h
151 / 324Feature importance caveats and class weighting effects
Importance rankings and weighted losses drive real decisions about features and thresholds. Both silently change meaning under correlation, cardinality and imbalance, so this unit teaches the distrust they deserve.
- explain
- measure
You can now
- Contrast impurity, permutation and gain-based importance and predict when each misleads.
- Measure impurity dilution under correlation and permutation stability across seeds on one dataset.
- Explain how class weighting and resampling move recall, thresholds and probability semantics, and when to recalibrate.
Covers
- Feature importance caveats under correlation and cardinality
- Class weighting and resampling effects on probability semantics
Study, in this order
5 steps
- sklearn forest feature importance evaluationscikit-learn - neighbors, SVMs, trees and ensembles
1.11.2.5 Feature importance evaluation: mean decrease in impurity, the train-only and high-cardinality flaws, the permutation alternative, and scores summing to 1.0
- HoML Chapter 6: feature importance in practiceHands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models
Book Chapter 6: Feature Importance (printed pp. 206-207). Notebook 06 cells: Feature Importance: the iris importance ranking loop and the MNIST pixel importance heatmap
- ISLP Chapter 8: random forests and variable importanceISLP - tree-based methods and support vector machines
Section 8.2.1, Variable Importance Measures with Figure 8.9 (pp. 345 to 346): mean-decrease-Gini importance; then Section 8.2.2 (pp. 346 to 347): the decorrelation rationale with \(m\) near \(\sqrt{p}\)
- sklearn SVM unbalanced problemsscikit-learn - neighbors, SVMs, trees and ensembles
1.4.1.3 Unbalanced problems:
class_weightmultipliers including thebalancedmode - HoML Chapter 3: precision and recall tradeoffHands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models
Book Chapter 3: The Precision/Recall Trade-Off (printed pp. 115-120). Notebook 03 cells: Precision/Recall Trade-off: retuning the decision threshold with
precision_scoreandrecall_scoreinstead of trusting 0.5
Focus on
- Impurity importance multiplies split impurity decrease by sample fraction, averages over trees and normalizes to 1. It is computed on train statistics and favors many-unique-value features, so treat it as a hypothesis, not a verdict.
- Correlated features break both rankings. With \(x_0\) uniform, a near-copy \(x_1 = x_0\) plus 0.01 noise, a noise column and the label \(x_0 > 0.5\), a 200-tree forest splits impurity importance 0.498 and 0.498, while held-out permutation gives \(x_0\) 0.511 and \(x_1\) 0.0. Every tree needs an \(x_0\) split to finish the boundary, so \(x_0\) rescues points that a shuffled \(x_1\) misroutes but not the reverse. Yet \(x_1\) alone still scores 0.995: zero permutation importance does not mean useless.
class_weightbalancedmultiplies by \(n/(2 \times \text{class count})\), here \(\left[\begin{smallmatrix}0.546&5.957\end{smallmatrix}\right]\) at 8.5 percent prevalence (two standard-normal features, label \(x_0 + x_1 + 0.5\,\text{noise} > 2\), \(n\) 4000, 30 percent test split, seed 0). Recall rises 0.64 to 0.95 while mean predicted probability doubles from 0.082 to 0.171 against a true rate of 0.086: weighting buys recall with miscalibration.- After weighting or resampling, 0.5 is no longer the neutral threshold. Retune it on held-out data or recalibrate before shipping decisions.
Reproduce closed-book
Traps
feature_importances_sums to 1.0, so ranks are relative: dropping a feature redistributes its mass and reshuffles the rest.- Permutation on train rewards overfit. Run it on held-out data with several repeats and report the spread, not just the mean.
- Resampling to 50/50 then deciding at 0.5 exports the sampler prior. Keep original priors, recalibrate, or move the threshold explicitly.
Practice
HoML Chapter 6: importance versus permutation reconstruction
Feature Importance section: rerun the iris RandomForestClassifier ranking loop, then add permutation_importance on held-out data with 10 repeats
Explain one disagreement between the two rankings from correlation or cardinality. Skip the MNIST heatmap; the iris contrast is the exercise.
Self-test
-
Twins read 0.498 and 0.498 in impurity but 0.511 and 0.0 in permutation. What happened?
Trees split interchangeably on either twin, so impurity mass halves across them. The label is exactly \(x_0 > 0.5\), so every tree also needs an \(x_0\) split near the boundary: shuffling \(x_0\) breaks those splits, while \(x_0\) repairs anything a shuffled \(x_1\) misroutes. Permutation measures loss given the other features, not usefulness alone; \(x_1\) by itself scores 0.995. -
What multipliers does
class_weightbalancedassign at 8.5 percent prevalence, and from what formula?About \(\left[\begin{smallmatrix}0.546&5.957\end{smallmatrix}\right]\) from \(\text{n\_samples}/(\text{n\_classes} \times \text{class count})\). The minority multiplier is roughly the inverse prevalence ratio. -
After weighting, mean predicted probability doubles while true prevalence stays fixed. Did accuracy necessarily improve, and what must change before deciding at 0.5?
Not necessarily. The weighted loss trades precision and calibration for recall, so accuracy can fall. Retune the threshold on held-out data or recalibrate the scores first. -
A noise feature reads 0.004 impurity against 0.0 permutation. Keep or drop it?
Neither on one number: 0.004 of a normalized train-only score against exactly 0.0 held-out permutation says it contributes nothing measurable. Drop it only after the ablated model holds its score.
Completion criterion
Stop when
Your demo matches the reference impurity and permutation split, your weighting rerun shows the recall gain with the calibration cost, and you can state the threshold policy for a weighted model.
Mark passed, then open 4.1 Only mark this when you can do it without notes.Leads to
- 5.8 Permutation importance and partial dependence Carry model-agnostic importance into evaluation.
- 5.5 Cost-sensitive thresholds, imbalance and prevalence change Set cost-sensitive thresholds with calibrated probabilities.
Comes after
- 3.2 Bagging, random forests and out-of-bag estimatesUse the forest you measured for importance caveats.
- 3.3 AdaBoost and gradient boostingAttribute the boosted model without fooling yourself.
- 3.6 Support vector machines: margins, hinge loss and slackTrace how \(C\) and class weights move decisions.
- 3.7 Kernels, bandwidth and kernel matrix costClose the section with interpretation and weighting caveats.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models Read Chapters 3, 5 and 6 and the online Appendix C notebook as the first pass through performance measures, trees, ensembles and SVMs, running notebooks 03, 05 and 06 alongside. Vary depth, estimators, learning rate, k, C and RBF width under fixed splits.
- scikit-learn - neighbors, SVMs, trees and ensembles Check official estimator constraints, complexity, preprocessing needs, missing-value behavior and probability caveats. Do not use library calls as the mechanism implementation.
- ISLP - tree-based methods and support vector machines Use Chapters 8 and 9 with their Python labs for the statistical account of trees, bagging, random forests, boosting and SVMs that the notebooks leave implicit, and for model-choice and feature-importance arguments.
Also in this section
- Deep-ML - tree, ensemble and kernel problems Use 138 Best Gini-Based Split, 20 Decision Tree Learning, 307 Bagging Classifier, 21 Pegasos Kernel SVM and 280 RBF Kernel as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including tie and update-order conventions.
- Probabilistic Machine Learning, Chapter 16: exemplar-based methods Read Sections 16.1 to 16.1.3 for the KNN decision rule, the curse of dimensionality with the growing-hypercube example, and the speed section on k-d trees, hashing and approximate search.
- XGBoost parameters and sklearn estimator interface Read the tree-booster parameter list for
eta, depth, sampling and regularization names and defaults, and the sklearn page for early stopping witheval_set. - LightGBM parameters Read
learning_rate,num_leaves,max_depth,min_data_in_leaf,feature_fraction,bagging_fractionwithbagging_freq,lambda_l2and early stopping defaults. - CatBoost common parameters and categorical features Read
iterations,learning_rate,l2_leaf_reg,depth,grow_policy,subsample,rsm,boosting_type,use_best_modelandone_hot_max_sizedefaults, plus native categorical handling.