XGBoost, LightGBM and CatBoost configuration

1. XGBoost tree-booster parameters

   XGBoost parameters and sklearn estimator interface

   Parameters for Tree Booster: eta default 0.3, max_depth 6, min_child_weight 1, subsample 1, colsample_bytree, bylevel and bynode 1, lambda 1, alpha 0

   Done
2. XGBoost sklearn interface and early stopping

   XGBoost parameters and sklearn estimator interface

   Early Stopping: XGBClassifier with tree_method hist, early_stopping_rounds with eval_set, best iteration used automatically for prediction

   Done
3. LightGBM core and control parameters

   LightGBM parameters

   learning_rate 0.1, num_leaves 31, max_depth -1 for no limit with leaf-wise growth, min_data_in_leaf 20, feature_fraction 1.0, bagging_fraction 1.0 effective only with bagging_freq above 0, lambda_l2 0.0, early stopping by callback or early_stopping_round

   Done
4. CatBoost common parameters

   CatBoost common parameters and categorical features

   depth 6, learning_rate chosen automatically for Logloss, MultiClass and RMSE and 0.03 otherwise, l2_leaf_reg 3.0, iterations 1000, SymmetricTree grow_policy, subsample 0.8 under the default MVS bootstrap on CPU (Bayesian for multiclass), rsm for columns per split, boosting_type Ordered versus Plain, use_best_model with eval_set, one_hot_max_size 2

   Done
5. sklearn histogram gradient boosting as the local baseline

   scikit-learn - neighbors, SVMs, trees and ensembles

   1.11.1.1 Histogram-Based Gradient Boosting with Usage: max_iter 100 replaces n_estimators, max_bins 255, max_leaf_nodes 31, early_stopping auto above 10000 rows

   Done
6. HoML Chapter 6: histogram boosting with native categories

   Hands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models

   Book Chapter 6: Histogram-Based Gradient Boosting (printed pp. 214-215). Notebook 06 cells: HistGradientBoostingRegressor with categorical_features=[0]: binned training with a native categorical slot

   Done

Focus on

• Learn the name map, in XGBoost, LightGBM and CatBoost order: step size is eta, learning_rate and learning_rate; rows are subsample, bagging_fraction and subsample; columns are colsample_bytree per tree, feature_fraction per tree and rsm per split (alias colsample_bylevel); L2 is lambda, lambda_l2 and l2_leaf_reg.
• Defaults that bite: XGBoost eta 0.3 against LightGBM 0.1, CatBoost 1000 iterations with a rate it picks itself for the common losses, LightGBM bagging off until bagging_freq passes 0, LightGBM growing leaf-wise to num_leaves 31 with no depth limit (max_depth -1), and CatBoost symmetric trees at depth 6.
• Early stopping differs per library: XGBoost takes early_stopping_rounds with eval_set and predicts from the best iteration, LightGBM takes early_stopping_round or the early_stopping callback with a validation set, CatBoost takes eval_set with use_best_model, and HGBT stops by itself only above 10000 rows.
• Categorical handling differs too: CatBoost takes raw categories with ordered target statistics and one-hot encodes only features with at most one_hot_max_size values (2 by default), LightGBM takes integer-coded categorical_feature columns, and XGBoost takes pandas category columns with enable_categorical=True on the hist or approx tree method.

Reproduce closed-book

Traps

• num_leaves 31 with max_depth 4 cannot grow 31 leaves: depth caps the tree at 16 leaves, so set both consistently instead of copying the default.
• bagging_fraction 0.8 alone changes nothing in LightGBM. Bagging stays off until bagging_freq is set above 0.
• Copying n_estimators 100 into CatBoost without setting the learning rate silently swaps the shrinkage regime, since CatBoost defaults to 1000 iterations with an automatic rate.

Practice

The section mini-project covers this: configured XGBoost, LightGBM and CatBoost runs with early stopping compared under fixed splits in the family-choice defense.

Self-test

1. What are the default step sizes in XGBoost, LightGBM and CatBoost?

   Show answer Got it Missed

2. Name the row-sampling parameter in each of the three libraries and any enabling switch.

   Show answer Got it Missed

3. Should low-cardinality string columns be one-hot encoded for CatBoost?

   Show answer Got it Missed

4. When does sklearn HGBT stop early without an explicit validation set?

   Show answer Got it Missed

Leads to

• 5.8 Permutation importance and partial dependence Explain the configured booster with model-agnostic tools.
• 5.7 Learning curves, error taxonomies and slices Diagnose the early-stopped runs with learning curves.

Comes after

• 3.3 AdaBoost and gradient boostingConfigure the production implementations of this loop.

Resources for this unit

• Hands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models
  Aurelien Geron (O'Reilly, 2025)primaryimplementation
  Read Chapters 3, 5 and 6 and the online Appendix C notebook as the first pass through performance measures, trees, ensembles and SVMs, running notebooks 03, 05 and 06 alongside. Vary depth, estimators, learning rate, k, C and RBF width under fixed splits.
• scikit-learn - neighbors, SVMs, trees and ensembles
  scikit-learnreferencereference
  Check official estimator constraints, complexity, preprocessing needs, missing-value behavior and probability caveats. Do not use library calls as the mechanism implementation.
• XGBoost parameters and sklearn estimator interface
  XGBoost developersreferencereference
  Read the tree-booster parameter list for eta, depth, sampling and regularization names and defaults, and the sklearn page for early stopping with eval_set.
• LightGBM parameters
  LightGBM developersreferencereference
  Read learning_rate, num_leaves, max_depth, min_data_in_leaf, feature_fraction, bagging_fraction with bagging_freq, lambda_l2 and early stopping defaults.
• CatBoost common parameters and categorical features
  CatBoost developersreferencereference
  Read iterations, learning_rate, l2_leaf_reg, depth, grow_policy, subsample, rsm, boosting_type, use_best_model and one_hot_max_size defaults, plus native categorical handling.