Bagging & Random Forests

Bagging trains on bootstrap samples

Bagging means bootstrap aggregating. Each tree is trained on a sample drawn with replacement from the training data. A random forest adds feature subsampling so each split considers only a subset of candidate features. Classification combines tree votes; regression averages tree predictions. The trees are deliberately made different, and aggregation often lowers variance compared with a single deep tree.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(
    n_estimators=300,
    max_features="sqrt",
    min_samples_leaf=2,
    class_weight="balanced",
    random_state=42,
    n_jobs=-1,
)

Tune diversity and capacity, not only the tree count

The number of trees controls ensemble size, but useful tuning also considers max_features, max_depth, min_samples_leaf and class weights. More trees reduce sampling noise until additional trees have diminishing returns. A forest is often robust on tabular problems, but probabilities can still require calibration and impurity-based feature importance can mislead when features differ in cardinality. Use held-out permutation importance for a performance-based view.