Interactions, feature selection and fit-state serialization

Features built outside the pipeline leak across splits and drift from serving code. One saved fitted object is the only representation that both training and serving run.

implement

You can now

Assemble numeric, categorical and interaction features in one ColumnTransformer with declared remainder behavior.
Run feature selection inside the split boundary and report the threshold with its validation effect.
Serialize the fitted pipeline with joblib and reload it to identical predictions, including unknown categories.

Covers

Interactions and feature selection inside the split boundary
Fit-state serialization and sparse-to-dense hazards

Study, in this order

3 steps

Interactions, custom transformers and pipelines
Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning

Book Chapter 2: Experiment with Attribute Combinations (printed pp. 68-69), then Custom Transformers and Transformation Pipelines (pp. 81-90). Notebook 02 cells: Experimenting with Attribute Combinations for ratio features; Custom Transformers with StandardScalerClone, RBF similarity and ClusterSimilarity on latitude and longitude; Transformation Pipelines with num_pipeline, cat_pipeline using handle_unknown=ignore, and the full ColumnTransformer

Depth · Skim the KNN-regressor transformer variant in Exercise 4; keep the ClusterSimilarity pattern and the pipeline assembly.

Done
`Pipeline` and `ColumnTransformer` contracts
scikit-learn - pipelines and composite estimators

8.1.1 Pipeline fit and transform chaining with the no-leakage rule and transformer caching; 8.1.4 ColumnTransformer for heterogeneous columns with remainder drop and fitted feature names

Done
Selection and persistence
Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning

Book Chapter 2: saving the model with joblib in Launch, Monitor, and Maintain Your System (printed pp. 100-101) and Exercise 3 (p. 104). Notebook 02 cells: Exercise 3 SelectFromModel with threshold=0.005 in the preparation pipeline; Model persistence using joblib with the production-script pattern

Depth · Treat the 0.005 threshold as a starting point, not an optimum: the notebook reports no gain there. Skip Exercises 1, 2 and 4 here.

Done

Focus on

Selection inside the pipeline is fit on training folds only. Selecting on all rows before cross-validation picks features with evaluation help.
ColumnTransformer drops unlisted columns by default under remainder=drop. Name every kept column or set remainder explicitly.
ClusterSimilarity and ratio features are fit state too: their centers and medians come from training rows and must serialize with the pipeline.
joblib saves the fitted object, not the build script. Reload in a fresh process and compare predictions before trusting the artifact.

Reproduce closed-book

Closed-book, sketch a ColumnTransformer with a median-plus-scaler numeric path and a most-frequent-plus-one-hot categorical path, state the remainder behavior, and write the three-line joblib save, load and predict-parity check.

Traps

Calling fit_transform on the test set with the same pipeline object. It refits every statistic on evaluation data; call transform only.
Tuning the SelectFromModel threshold on the test score. The notebook run at 0.005 shows no gain; tune it under cross-validation or hold it fixed.
Saving the model but rebuilding preprocessing by hand in serving. Any median, category list or cluster center that differs silently shifts every prediction.

Practice

Hands-On Machine Learning Ch2 Exercise 3: `SelectFromModel` pipeline

Exercise solutions, Exercise 3: add SelectFromModel after the preparation pipeline and evaluate

Sweep the threshold under cross-validation starting from 0.005, and keep the fitted pipeline plus the threshold in the saved artifact.

Hands-On Machine Learning Ch2 Exercise 5: preparation search

Exercise solutions, Exercise 5: explore preparation options with RandomizedSearchCV

Run on a small subsample. Record which preparation choice moves the validation score and keep the search inside the training split.

Self-test

Where does SelectFromModel sit relative to the cross-validation split, and what breaks if it sits outside?
A new column appears in serving data. What does ColumnTransformer do with it?
A joblib reload matches old predictions, but new-category rows differ from training time. Is the artifact broken?

Completion criterion

Stop when

A fresh process loads your joblib pipeline, reproduces training predictions exactly, transforms unseen categories without refitting, and your threshold choice cites a cross-validated number.

Mark passed, then open 4.3 Only mark this when you can do it without notes.

Leads to

4.3 PCA via SVD: directions, explained variance and sign ambiguity Feed the fitted pipeline output into PCA and read reconstruction against the same split.
2.2 Conditioning, basis expansions, ridge, lasso and elastic net Connect scaling and basis choices to optimizer geometry and penalties.

Comes after

4.1 Numeric transforms and categorical encodingCarry the fitted encoders and scalers into pipelines with selection and saved state.

Resources for this unit

Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning
Aurelien Geron (O'Reilly, 2025)primarycourse
Read Chapters 7 and 8 and run notebooks 07 and 08 for PCA, k-means, DBSCAN, agglomerative clustering, Gaussian mixtures and anomaly detection. Stop before extended semi-supervised and face-data projects.
scikit-learn - pipelines and composite estimators
scikit-learnreferencereference
Read 8.1.1 Pipeline chaining with the no-leakage rule and transformer caching, and 8.1.4 ColumnTransformer for heterogeneous columns with remainder and fitted-state behavior. Use it to assemble and serialize the feature-selection pipeline.

Also in this section

scikit-learn - decomposition, clustering and outlier detection
scikit-learnreferencereference
Check decomposition orientation and solver behavior, cluster API semantics, and the distinction between novelty and outlier detection after the independent tests pass.
ISLP - Unsupervised Learning
James, Witten, Hastie, Tibshirani and Taylorrequiredtheory
Use Chapter 12 and its Python lab for the PCA derivations (12.2, including the low-rank approximation view and proportion of variance explained) and k-means (12.4.1), plus stability and interpretation limits, beside the notebook route.
Deep-ML - unsupervised problems
Deep-MLsupportpractice
Use 17 K-Means Clustering, 19 PCA, 341 Gaussian Mixture Model with EM, 259 DBSCAN and 367 Isolation Forest as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including the 367 path-length convention.
scikit-learn - preprocessing data
scikit-learnreferencereference
Read 8.3.1 standardization with the sparse with_mean=False rule, 8.3.1.3 scaling data with outliers, and 8.3.4 categorical encoding with handle_unknown, infrequent and sparse_output behavior. Apply after the HOML notebook build works.
scikit-learn - imputation of missing values
scikit-learnreferencereference
Read 8.4.2 univariate median and most_frequent strategies, 8.4.6 Marking imputed values with MissingIndicator or add_indicator=True for missingness flags, and 8.4.7 estimators that accept NaN directly. Decide impute-only versus impute-plus-indicator per column.
UMAP - basic usage and parameters
McInnes, Healy and the UMAP contributorsreferencereference
Work the Penguin scaling and fit_transform section and the Digits fit and transform section, then read the parameters page for the n_neighbors local-global tradeoff and min_dist packing. Stay in 2D exploratory views with fixed seeds.