Unit 3.3 · about 5 h
146 / 324AdaBoost and gradient boosting
Boosting turns weak rules into strong models by fitting what is left over. The same loop powers the gradient boosting libraries, so its failure modes of noise chasing and late overfitting follow you into production.
- derive
- implement
You can now
- Derive the AdaBoost reweighting step of error rate, learner weight and distribution update, and run one round by hand.
- Derive pseudo-residual fitting for squared and logistic loss and explain shrinkage, interaction depth and early stopping.
- Implement stage-wise boosting around stumps, match sklearn on a toy, and diagnose noise sensitivity and overfitting by iteration count.
Covers
- AdaBoost sample weighting and weak learners
- Gradient boosting, pseudo-residuals, shrinkage and early stopping
Study, in this order
4 steps
- HoML Chapter 6: AdaBoost and gradient boostingHands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models
Book Chapter 6: AdaBoost and Gradient Boosting (printed pp. 207-214). Notebook 06 cells: AdaBoost with the step-by-step weight update cells, Gradient Boosting with the residual-fitting chain and the
learning_rate1.0 versus 0.05 comparisonDepth · Drop
algorithm='SAMME'from the AdaBoost cell: sklearn 1.9 removed the argument and raisesTypeError, and SAMME is now the only mode. - ISLP Chapter 8: boosting regression treesISLP - tree-based methods and support vector machines
Section 8.2.3 with Algorithm 8.2 and the three tuning parameters \(B\), shrinkage \(\lambda\) and interaction depth \(d\) (pp. 347 to 350)
- sklearn AdaBoost usagescikit-learn - neighbors, SVMs, trees and ensembles
1.11.7 AdaBoost with 1.11.7.1 Usage: SAMME for classification, default decision stumps, and
n_estimatorswithlearning_rateas the main knobs - sklearn gradient boosting: loss, shrinkage and subsamplingscikit-learn - neighbors, SVMs, trees and ensembles
1.11.1.2.5 Shrinkage via learning rate (
learning_rateat or below 0.1 with largen_estimators) and 1.11.1.2.6 Subsampling (subsample0.5, OOB pessimistic so prefer cross-validation)
Focus on
- AdaBoost scores the round by the weighted error \(r\), pays the learner \(\alpha = \text{learning\_rate} \times \ln((1 - r)/r)\), multiplies wrong-point weights by \(\exp(\alpha)\) and renormalizes. The hand round gives \(r = 0.25\), \(\alpha = 0.5493\) and weights \(\left[\begin{smallmatrix}0.2113&0.2113&0.366&0.2113\end{smallmatrix}\right]\), matching
estimator_weights_. - Gradient boosting fits each tree to pseudo-residuals, the negative gradients of the loss. For squared loss those are ordinary residuals, and the target mean plus a depth-2 tree fit on the residuals exactly equals a one-tree GBM at learning rate 1.0.
- Shrinkage, depth and count interact: small \(\lambda\) needs large \(B\) with typical values 0.01 or 0.001, depth-1 stumps give additive single-variable terms, and
subsample0.5 adds bagging noise. - Boosting uses no bootstrap and each tree depends on its predecessors, so \(B\) is a genuine overfitting knob. Choose it by validation or early stopping, unlike bagging counts.
Reproduce closed-book
X = [0, 1, 2, 3] with y = [0, 1, 0, 1] and learning_rate 0.5: stump threshold 0.5 (2.5 ties; sklearn picks 0.5), \(r = 0.25\), \(\alpha = 0.5493\), new weights \(\left[\begin{smallmatrix}0.2113&0.2113&0.366&0.2113\end{smallmatrix}\right]\). Confirm \(\alpha\) against AdaBoostClassifier estimator_weights_. Then show GradientBoostingRegressor with learning_rate 1.0, one tree and max_depth 2 equals the target mean plus a depth-2 DecisionTreeRegressor fit on the residuals, on 50 points with \(x\) uniform on \([0, 4)\) and \(y = \sin(x)\) plus noise 0.05 at seed 0.Traps
- Copying
algorithm='SAMME'from the notebook raisesTypeErrorin sklearn 1.9. The parameter is gone and discrete SAMME is the only behavior. - AdaBoost pushes weight exponentially onto mistakes, so mislabeled points attract ever more attention. Noisy labels hurt it more than they hurt bagging.
- The
learning_rate1.0 residual demo is didactic. Real runs shrink to 0.1 or below with many more trees, or the residual fit jumps past good solutions.
Practice
ISLP Chapter 8, Applied exercise 10: boosting on Hitters salary
Exercise 10 (p. 366), parts (a) to (g): 1000 trees over a shrinkage range, train and test MSE plots, comparison with Chapters 3 and 6 regressions and with bagging, variable importance
Install the ISLP package for the Hitters data, drop the rows with missing Salary, log-transform salary, use the first 200 rows for training, and report the shrinkage value with the lowest test MSE.
Self-test
-
A stump has weighted error 0.25 with
learning_rate0.5. What learner weight does AdaBoost assign, and what happens to a wrong point starting at 0.25?\(\alpha = 0.5 \ln(3) = 0.5493\). The wrong point goes to \(0.25 \times \exp(0.5493) = 0.433\) before renormalization, which is 0.366 after. -
Why does boosting depth-1 stumps produce an additive model?
Each stump splits on one variable, so each term is a function of one variable only. The sum over rounds is a sum of single-variable terms. -
A GBM with 5000 trees,
learning_rate0.5 and no validation set scores perfectly in training but poorly on new data. What broke and what is the fix?Late trees overfit the residuals. Shrink the learning rate toward 0.1 or below, add row or column subsampling, and pick the iteration count by early stopping on validation. -
How does boosting reuse data differently from bagging?
Bagging fits independent trees on bootstrap copies. Boosting refits the same data with modified weights or residuals, so each tree depends strongly on the ones before it.
Completion criterion
Stop when
Your hand round matches sklearn weights, your stump booster reproduces the one-tree GBM identity, and the Hitters sweep shows test MSE dipping then rising with a stated choice of shrinkage and tree count.
Mark passed, then open 3.4 Only mark this when you can do it without notes.Leads to
- 3.4 XGBoost, LightGBM and CatBoost configuration Configure the production implementations of this loop.
- 3.8 Feature importance caveats and class weighting effects Attribute the boosted model without fooling yourself.
Comes after
- 3.1 CART: split objectives, impurity, stopping and pruningFit the same splits to residuals and reweighted data.
- 3.2 Bagging, random forests and out-of-bag estimatesContrast parallel bagging with sequential boosting.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models Read Chapters 3, 5 and 6 and the online Appendix C notebook as the first pass through performance measures, trees, ensembles and SVMs, running notebooks 03, 05 and 06 alongside. Vary depth, estimators, learning rate, k, C and RBF width under fixed splits.
- scikit-learn - neighbors, SVMs, trees and ensembles Check official estimator constraints, complexity, preprocessing needs, missing-value behavior and probability caveats. Do not use library calls as the mechanism implementation.
- ISLP - tree-based methods and support vector machines Use Chapters 8 and 9 with their Python labs for the statistical account of trees, bagging, random forests, boosting and SVMs that the notebooks leave implicit, and for model-choice and feature-importance arguments.
Also in this section
- Deep-ML - tree, ensemble and kernel problems Use 138 Best Gini-Based Split, 20 Decision Tree Learning, 307 Bagging Classifier, 21 Pegasos Kernel SVM and 280 RBF Kernel as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including tie and update-order conventions.
- Probabilistic Machine Learning, Chapter 16: exemplar-based methods Read Sections 16.1 to 16.1.3 for the KNN decision rule, the curse of dimensionality with the growing-hypercube example, and the speed section on k-d trees, hashing and approximate search.
- XGBoost parameters and sklearn estimator interface Read the tree-booster parameter list for
eta, depth, sampling and regularization names and defaults, and the sklearn page for early stopping witheval_set. - LightGBM parameters Read
learning_rate,num_leaves,max_depth,min_data_in_leaf,feature_fraction,bagging_fractionwithbagging_freq,lambda_l2and early stopping defaults. - CatBoost common parameters and categorical features Read
iterations,learning_rate,l2_leaf_reg,depth,grow_policy,subsample,rsm,boosting_type,use_best_modelandone_hot_max_sizedefaults, plus native categorical handling.