Unit 3.2 · about 3 h
145 / 324Bagging, random forests and out-of-bag estimates
Single trees have high variance: the same algorithm learns visibly different models from the same data. Bagging plus feature subsampling is the standard fix, and out-of-bag scoring gives a nearly free error estimate.
- explain
- measure
You can now
- Explain why bagging cuts variance for deep trees and why the error stabilizes as the tree count grows instead of overfitting.
- Explain how random feature subsets decorrelate trees and how
max_featurestrades correlation against strength. - Measure out-of-bag error against held-out error and read
n_estimators,max_featuresandoob_scoresettings in sklearn.
Covers
- Bagging, bootstrap samples and variance reduction
- Random forests, feature subsampling and out-of-bag estimates
Study, in this order
4 steps
- HoML Chapter 6: voting, bagging and forestsHands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models
Book Chapter 6, from Voting Classifiers through Extra-Trees (printed pp. 196-206). Notebook 06 cells: Voting Classifiers, Bagging and Pasting in Scikit-Learn, Out-of-Bag evaluation with the
exp(-1)cells, Random Forests, Extra-Trees - ISLP Chapter 8: bagging, forests and the ensemble summaryISLP - tree-based methods and support vector machines
Sections 8.2.1, 8.2.2 and 8.2.5 (pp. 343 to 347, and 353): bootstrap aggregation, OOB as a valid test-error estimate with around \(B/3\) predictions per point, and \(m\) near \(\sqrt{p}\)
- sklearn forests: parameters and parallelizationscikit-learn - neighbors, SVMs, trees and ensembles
Random Forests through Parameters (
max_featuressqrtfor classifiers versus 1.0 for regressors,bootstrapTrueversusExtraTreesFalse) and Parallelization withoob_score - sklearn bagging meta-estimatorscikit-learn - neighbors, SVMs, trees and ensembles
1.11.3 Bagging: bootstrap versus pasting,
max_samples,max_featuresandoob_score
Focus on
- A bootstrap covers about 63.2 percent of unique points since \((1 - 1/n)^n\) tends to \(1/e\). Each point is predicted by the trees that missed it. On
make_moons(2000, noise 0.25, seed 0)with a 25 percent test split, 200 trees give OOB 0.934 against held-out 0.932. - Bagging helps unstable learners because averaging kills variance while bias stays put. Adding trees stabilizes the error; unlike boosting it has no residual-chasing loop to overfit with.
- Forests force different top splits by sampling \(m\) near \(\sqrt{p}\) predictors per split, which decorrelates trees that would otherwise copy one strong predictor.
- sklearn defaults differ by task: the regressor uses
max_features1.0, which is plain bagging, andExtraTreesusesbootstrapFalsewith random splits.
Reproduce closed-book
RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0) on make_moons(n_samples=2000, noise=0.25, random_state=0) split with test_size=0.25 and random_state=0, and show OOB 0.934 against held-out 0.932.Traps
- Treating OOB as free model selection. Each OOB prediction uses only about a third of the trees, so with few trees it reads pessimistic, and once you tune on it, it is no longer held out. Monitor with it; select with a validation split or cross-validation.
- Soft voting needs
predict_probafrom every member. A defaultSVCin sklearn 1.9 has none, and the softVotingClassifierstill fits; theAttributeErrorappears only at predict time. - Assuming
sqrtfeatures for a regressor silently turns the forest into bagging. Readmax_featuresper estimator instead of quoting the classifier rule.
Practice
Deep-ML 307: Implement Bagging Classifier from Scratch
Implement bagging_classifier with decision stumps and majority vote; the visible tests check [0 1] with 5 stumps at seed 42 and [0 1 1] with 10 stumps at seed 0
Bootstrap \(n\) samples with replacement per stump, minimize stump misclassification error, combine by majority vote. The statement sets no even-vote tie rule, so keep yours explicit and match the visible outputs.
Self-test
-
From \(n = 200\) points, how many land out of bag and how many trees predict each one?
About \(0.368 \times 200 = 74\) points are missed per tree, and each point is predicted by the roughly \(B/3\) trees whose samples skipped it. -
Ten bagged trees give red probabilities 0.1, 0.15, 0.2, 0.2, 0.55, 0.6, 0.6, 0.65, 0.7 and 0.75. What do majority vote and probability averaging predict?
Majority vote says red with 6 of 10 above 0.5. Averaging gives 0.45 and says green. The combination rule changes the answer. -
Is a default RandomForestRegressor closer to a sqrt forest or to bagged trees?
Bagged trees, because itsmax_featuresdefault is 1.0. Thesqrtdefault applies to classifiers only. -
Why does raising \(B\) not overfit bagging while it can overfit boosting?
Bagging averages fixed-complexity trees fit on resamples, so variance shrinks and bias holds. Boosting keeps fitting residuals, so late trees can chase noise; choose \(B\) by validation.
Completion criterion
Stop when
You derive the 0.368 fraction, your moons OOB lands within about a point of held-out error, and problem 307 passes.
Mark passed, then open 3.3 Only mark this when you can do it without notes.Leads to
- 3.3 AdaBoost and gradient boosting Contrast parallel bagging with sequential boosting.
- 3.8 Feature importance caveats and class weighting effects Use the forest you measured for importance caveats.
Comes after
- 3.1 CART: split objectives, impurity, stopping and pruningGrow forests from the trees you just built.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models Read Chapters 3, 5 and 6 and the online Appendix C notebook as the first pass through performance measures, trees, ensembles and SVMs, running notebooks 03, 05 and 06 alongside. Vary depth, estimators, learning rate, k, C and RBF width under fixed splits.
- scikit-learn - neighbors, SVMs, trees and ensembles Check official estimator constraints, complexity, preprocessing needs, missing-value behavior and probability caveats. Do not use library calls as the mechanism implementation.
- ISLP - tree-based methods and support vector machines Use Chapters 8 and 9 with their Python labs for the statistical account of trees, bagging, random forests, boosting and SVMs that the notebooks leave implicit, and for model-choice and feature-importance arguments.
Also in this section
- Deep-ML - tree, ensemble and kernel problems Use 138 Best Gini-Based Split, 20 Decision Tree Learning, 307 Bagging Classifier, 21 Pegasos Kernel SVM and 280 RBF Kernel as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including tie and update-order conventions.
- Probabilistic Machine Learning, Chapter 16: exemplar-based methods Read Sections 16.1 to 16.1.3 for the KNN decision rule, the curse of dimensionality with the growing-hypercube example, and the speed section on k-d trees, hashing and approximate search.
- XGBoost parameters and sklearn estimator interface Read the tree-booster parameter list for
eta, depth, sampling and regularization names and defaults, and the sklearn page for early stopping witheval_set. - LightGBM parameters Read
learning_rate,num_leaves,max_depth,min_data_in_leaf,feature_fraction,bagging_fractionwithbagging_freq,lambda_l2and early stopping defaults. - CatBoost common parameters and categorical features Read
iterations,learning_rate,l2_leaf_reg,depth,grow_policy,subsample,rsm,boosting_type,use_best_modelandone_hot_max_sizedefaults, plus native categorical handling.