Section 4 of 5 · Classical Machine Learning

Features, Representation & Unsupervised Learning

Preprocessing and representation alter the geometry seen by every estimator. Unsupervised output becomes useful evidence only when its stability and downstream value are tested.

8 units 17 topics 0/8

Units

8 in this section

4.1 Numeric transforms and categorical encoding
implement
4 h
4.2 Interactions, feature selection and fit-state serialization
implement
4 h
4.3 PCA via SVD: directions, explained variance and sign ambiguity
derive · implement
5 h
4.4 k-means: objective, initialization, empty clusters and instability
derive · implement
4 h
4.5 Gaussian mixtures and EM
derive
3 h
4.6 DBSCAN, agglomerative clustering and cluster validation
explain · measure
4 h
4.7 Outlier and novelty detection, thresholds and representation
explain · measure
3 h
4.8 t-SNE and UMAP as guarded exploratory views
explain
2 h

Before moving on

Checkpoint

Produce a reproducible feature, PCA and clustering result with invariant-aware tests, then state what the clusters or anomaly scores do and do not justify from stability and downstream evidence.

Deliberately out of scope

Leave for later

Deep representation learning
Full manifold-learning theory
Semantic claims from unlabeled clusters

Resources for this section

Shared by every unit above

Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning
Aurelien Geron (O'Reilly, 2025)course

primary

Read Chapters 7 and 8 and run notebooks 07 and 08 for PCA, k-means, DBSCAN, agglomerative clustering, Gaussian mixtures and anomaly detection. Stop before extended semi-supervised and face-data projects.
scikit-learn - decomposition, clustering and outlier detection
scikit-learnreference

reference

Check decomposition orientation and solver behavior, cluster API semantics, and the distinction between novelty and outlier detection after the independent tests pass.
ISLP - Unsupervised Learning
James, Witten, Hastie, Tibshirani and Taylortheory

required

Use Chapter 12 and its Python lab for the PCA derivations (12.2, including the low-rank approximation view and proportion of variance explained) and k-means (12.4.1), plus stability and interpretation limits, beside the notebook route.
Deep-ML - unsupervised problems
Deep-MLpractice

support

Use 17 K-Means Clustering, 19 PCA, 341 Gaussian Mixture Model with EM, 259 DBSCAN and 367 Isolation Forest as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including the 367 path-length convention.
scikit-learn - preprocessing data
scikit-learnreference

reference

Read 8.3.1 standardization with the sparse with_mean=False rule, 8.3.1.3 scaling data with outliers, and 8.3.4 categorical encoding with handle_unknown, infrequent and sparse_output behavior. Apply after the HOML notebook build works.
scikit-learn - pipelines and composite estimators
scikit-learnreference

reference

Read 8.1.1 Pipeline chaining with the no-leakage rule and transformer caching, and 8.1.4 ColumnTransformer for heterogeneous columns with remainder and fitted-state behavior. Use it to assemble and serialize the feature-selection pipeline.
scikit-learn - imputation of missing values
scikit-learnreference

reference

Read 8.4.2 univariate median and most_frequent strategies, 8.4.6 Marking imputed values with MissingIndicator or add_indicator=True for missingness flags, and 8.4.7 estimators that accept NaN directly. Decide impute-only versus impute-plus-indicator per column.
UMAP - basic usage and parameters
McInnes, Healy and the UMAP contributorsreference

reference

Work the Penguin scaling and fit_transform section and the Digits fit and transform section, then read the parameters page for the n_neighbors local-global tradeoff and min_dist packing. Stay in 2D exploratory views with fixed seeds.

Apply the section

Mini-project · Feature pipeline, PCA and clustering with invariant-aware tests (10 to 14 h)

Build a train-only feature pipeline, implement PCA and k-means in NumPy with tests that respect sign and label-permutation invariance, then decide from stability and downstream evidence what the components, clusters or anomaly scores do and do not justify.

Steps

Build a ColumnTransformer pipeline with imputation plus missingness indicators, scaling and one-hot encoding with handle_unknown, fitted on train only. Add a schema test that fails on unexpected columns, and count unknown categories per batch.
Implement PCA through the SVD of the centered data: components, explained variance ratio, projection and reconstruction. Compare with scikit-learn PCA through sign-invariant quantities.
Implement k-means with k-means++ initialization. Test that inertia never rises across iterations, and compare with KMeans from the same initial centers through the adjusted Rand index, which ignores label order.
Add one more method with its own failure mode: a DBSCAN eps sweep on moons, or a Gaussian mixture with its covariance type and reg_covar stated.
Measure stability: rerun clustering over seeds and bootstrap resamples and report the adjusted Rand distribution. Choose \(k\) from stability and silhouette together, noting silhouette's bias toward convex clusters.
Test downstream value: feed the components or cluster ids to a supervised model on a fixed split and report the paired change.
Fit an anomaly detector on clean data (IsolationForest, or LOF with novelty=True), set the threshold from a stated flag-rate budget, and report the flag rate on a shifted batch with counts and slices.

Deliverables

The pipeline package with its fit-state and schema tests.
PCA and k-means implementations with sign- and permutation-invariant tests.
The evidence report: stability distributions, the choice of k, the downstream effect, the anomaly threshold and a closing paragraph on what the results do and do not justify.

Checks

external-tests · Deep-ML 17, 19, 341, 259 and 367
rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Fit state	All learned state comes from train only, and a schema test catches unexpected columns.	Unknown categories are counted per batch with an alert rule, and indicator columns survive `get_feature_names_out`.
PCA	Components and explained variance match scikit-learn up to sign.	Tests assert only sign-invariant quantities, and they pass across solvers.
Clustering	k-means matches scikit-learn from the same initial centers (adjusted Rand index 1 on separated blobs), and inertia never rises.	Tests are label-permutation invariant, and the second method's failure mode is shown and explained.
Stability and claims	Stability over seeds and resamples is reported with the chosen \(k\).	A downstream test shows whether the clusters or components help, and the report states what they do not justify.
Anomaly threshold	The threshold follows from a stated flag budget on clean data.	The flag-rate change on the shifted batch is reported with counts and slices before any threshold change.

Feeds into: Milestone artifact 2, the mechanisms suite: PCA and k-means come from here. The stability evidence feeds the decision dossier.