Decision Trees

Understand how decision trees partition feature space and how depth and leaf constraints control overfitting.

By the end of this lesson

You will be able to

Describe a split, branch, leaf and prediction.
Explain impurity reduction as a split-selection objective.
Distinguish shallow and deep trees.
Use validation performance to select complexity controls.

A tree asks a sequence of questions

A decision tree repeatedly splits training rows using feature tests. A classification split aims to create child groups with purer class labels. Common impurity measures include Gini impurity and entropy. A leaf returns a class or class probability based on the training rows that reach it. A tree can model nonlinear boundaries and interactions without manually specifying them.

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(
    max_depth=4,
    min_samples_leaf=20,
    random_state=42,
)
tree.fit(X_train_encoded, y_train)

Complexity controls generalization

A tree that keeps splitting can memorize training examples. max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes and cost-complexity pruning limit tree complexity. Compare shallow and deeper trees on training and validation scores: a large gap often signals overfitting. Trees usually do not require standardized numeric features, but categorical values still need appropriate encoding.

Apply what you learned

Exercise

Inspect tree complexity

Fit decision trees with different depth and leaf constraints and compare training and validation behavior.

Create a reproducible train/validation split.
Fit a shallow tree and a less-constrained tree on the same training rows.
Compare training and validation metrics, not training accuracy alone.
Inspect the top-level splits and one path to a leaf.
Choose a complexity setting and explain underfitting or overfitting.

Done when

The chosen tree is evaluated on held-out rows, uses explicit complexity controls and explains the gap between training and validation performance.

Free references

Study resources

2 links

01
Decision and Classification Trees, Clearly Explained Video
Watch the individual decision-tree explanation. Focus on Gini impurity, split choice and overfitting.

Done
02
Decision Trees Article
Read the classification-tree overview and the section on controlling tree size.

Done

Check your understanding

Self-test

2 questions

Question 1

What does a leaf node represent?
Question 2

Name two controls that can reduce tree overfitting.

Completion check

Move on when

You can train a tree, explain its split logic and select complexity using held-out performance.

Mark passed and continue Back to module