Cross-Validation & Hyperparameter Search

Estimate model performance across folds and tune hyperparameters using a split strategy that matches the observations.

By the end of this lesson

You will be able to

Distinguish model parameters from hyperparameters.
Explain k-fold and stratified cross-validation.
Choose grouped or time-aware cross-validation when needed.
Use GridSearchCV or RandomizedSearchCV with a task-relevant metric.

Cross-validation repeats the fit on different folds

In k-fold cross-validation, data is divided into k folds. Each fold is held out once while the remaining folds train the model. The scores summarize how the procedure performs across several partitions. Stratified k-fold keeps class proportions approximately stable for classification. Grouped folds keep related observations together, while time-aware splits preserve temporal order. The split strategy must match how new predictions will be encountered.

from sklearn.model_selection import StratifiedKFold, cross_validate

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(
    model, X_train, y_train,
    cv=cv,
    scoring=["average_precision", "f1", "roc_auc"],
)

Search inside the training boundary

A hyperparameter is a setting chosen around model fitting, such as tree depth, regularization strength or neighbor count. GridSearchCV tests a specified grid; RandomizedSearchCV samples a fixed number of combinations. Give the search a scoring metric and cross-validation strategy that fit the task. If preprocessing is inside the Pipeline, every fold fits transformations independently. Run the search only on training data; use validation data for any later threshold choice, and reserve test data for final evaluation.

Apply what you learned

Exercise

Run a controlled parameter search

Tune one model's hyperparameters with cross-validation while keeping the final test partition untouched.

Select an estimator and define a small, justified parameter grid or distribution.
Wrap learned preprocessing and the estimator in one Pipeline.
Choose a suitable cross-validation strategy and scoring metric.
Run GridSearchCV or RandomizedSearchCV on training rows only.
Record best parameters, mean fold score and variation; evaluate the selected model on test once.

Done when

Search fits only on training data, preprocessing is inside the pipeline, CV matches the data structure and test data is used only after selection.

Free references

Study resources

2 links

01
Cross-validation: Evaluating Estimator Performance Article
Read the overview and stratified, grouped and time-aware cross-validation sections.

Done
02
Tuning the Hyper-parameters of an Estimator Article
Read GridSearchCV and RandomizedSearchCV, including scoring and cross-validation arguments.

Done

Check your understanding

Self-test

2 questions

Question 1

What does cross-validation estimate?
Question 2

Where should hyperparameter search occur?

Completion check

Move on when

You can tune a pipeline using a suitable cross-validation strategy and keep the final test set out of model selection.

Mark passed and continue Back to module