Data Splits, Baselines & Leakage

Separate model fitting from model selection and final evaluation, and identify leakage before it inflates a metric.

By the end of this lesson

You will be able to

Explain the purpose of each data split.
Choose random, stratified, grouped or time-aware splitting based on the data-generating process.
Detect target leakage and preprocessing leakage.
Compare fitted models with a simple baseline.

Each split has a separate job

The training set estimates model parameters. The validation set helps compare models, tune hyperparameters and choose decision thresholds. The test set estimates performance after those choices are complete. Cross-validation can replace a single validation fold when data is limited, but a final test set should still be reserved. Stratified splits preserve class proportions for independent examples; group-aware splits keep related people or devices together; time-aware splits preserve temporal order. The strategy should match how new predictions will arrive.

Leakage makes the score dishonest

Leakage occurs when information unavailable at prediction time, or information from a held-out partition, influences a fitted model. Examples include scaling the whole dataset before splitting, selecting features after looking at test labels, or using a post-outcome column such as Bank Marketing duration. Fit preprocessing on training rows only. Compare against baselines such as DummyClassifier; a sophisticated model that cannot beat a sensible baseline has not demonstrated value.

from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
baseline = DummyClassifier(strategy="prior")
baseline.fit(X_train, y_train)
print(baseline.score(X_test, y_test))

Apply what you learned

Exercise

Audit a split for leakage

Create a split plan for a binary classification dataset and identify which operations must be fitted on training rows only.

Choose a splitting strategy and justify it from how observations were collected.
Create train, validation and test partitions without using test labels for selection.
Calculate class proportions for each partition.
Build a majority-class baseline and report one class-aware metric.
Identify three potential leakage paths, including the duration column in Bank Marketing.

Done when

The split plan respects the data-generating process, the baseline is evaluated on held-out rows, and every learned preprocessing operation is fitted only on training data.

Free references

Study resources

2 links

01
Common Pitfalls and Recommended Practices Article
Read data leakage and inconsistent preprocessing. Note when to use fit_transform versus transform.

Done
02
Cross-validation: Evaluating Estimator Performance Article
Read the introduction and compare stratified, grouped and time-aware cross-validation.

Done

Check your understanding

Self-test

2 questions

Question 1

What is the test set for?
Question 2

Can preprocessing leak information without using the target?

Completion check

Move on when

You can justify the split strategy, build a baseline and show that no test or post-outcome information enters model fitting.

Mark passed and continue Back to module