Unit 4.3 · about 5 h
154 / 324PCA via SVD: directions, explained variance and sign ambiguity
PCA is the baseline representation for wide tabular data and the preprocessing step inside the clustering and anomaly cards. Its signs are arbitrary, so any test that hard-codes them is wrong.
- derive
- implement
You can now
- Derive PCA projection, explained variance and reconstruction error from the SVD of the centered matrix.
- Implement PCA from a blank file and match scikit-learn up to sign flips and component order.
- Choose the component count from cumulative explained variance and reconstruction cost, and write tests invariant to sign.
Covers
- Centered PCA, SVD and principal directions
- Explained variance, reconstruction error and component choice
- PCA sign ambiguity and invariant tests
Study, in this order
3 steps
- PCA from SVD to model selectionHands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning
Book Chapter 7: PCA, from Preserving the Variance through Incremental PCA (printed pp. 227-236). Notebook 07 cells: PCA through Choosing the Right Number of Dimensions: manual SVD of
X_centeredasUSVt, Using Scikit-Learn, Explained Variance Ratio, thecumsumrule for the component count; PCA for Compression; Randomized PCA withsvd_solverand Incremental PCADepth · Skip Random Projection, LLE and Extra Material Kernel PCA. They return in the visualization unit or later work.
- PCA orientation and solversscikit-learn - decomposition, clustering and outlier detection
2.5.1 PCA: centering without scaling,
components_orientation and thewhitenoption; 2.5.1.3 randomized SVD cost and the inexactinverse_transformnote; 2.5.1.2IncrementalPCApartial_fitbehavior - PCA lab with scaling and signISLP - Unsupervised Learning
Principal Components Analysis on USArrests: scaling before PCA, scores via
transform, the biplot with its y-axis reflection note, standard deviations and PVE with the scree plot
Focus on
- PCA centers but does not scale. Features in larger units dominate the first components; standardize first when units differ.
- Component variance is \(s_j^2/(n-1)\); the ratio divides by the total. Randomized solvers approximate both.
- Component signs are arbitrary: flipping a component with its scores leaves every reconstruction unchanged. Tests must use absolute values or reconstruction error.
- The cumsum rule at 0.95 is a budget, not an optimum. Downstream error and cost decide the count.
Reproduce closed-book
Traps
- Hard-coding component signs from one run. Sign conventions differ across libraries, solvers and versions; assert with
absor compare reconstructions. - Calling
PCA(n_components=0.95)a fixed dimension. It selects whatever count reaches 95 percent on the fitted data; refitting on new data can change the width. - Using randomized-solver
inverse_transformas exact. The docs note it is not the exact inverse oftransformeven withwhiten=False.
Practice
Deep-ML 19: Principal Component Analysis Implementation
Problem 19: implement pca(X, k) with standardization, covariance, eigh and the stated sign rule
The two tests check \(k=1\) components \(\left[\begin{smallmatrix}0.7071\\0.7071\end{smallmatrix}\right]\) and \(\left[\begin{smallmatrix}0.7071\\-0.7071\end{smallmatrix}\right]\) on small matrices. Match the sign rule exactly: the first element with absolute value above 1e-10 is made positive.
Self-test
-
Why do \(\left[\begin{smallmatrix}1&2\\3&4\\5&6\end{smallmatrix}\right]\) yield ratio 1.0 on one component?
Centered rows are multiples of \(\left[\begin{smallmatrix}1&1\end{smallmatrix}\right]\), so the matrix has rank one. The single singular value 4 carries all variance. Verified with NumPy:s=[4,0]. -
Two runs give components \(\left[\begin{smallmatrix}0.7&0.7\end{smallmatrix}\right]\) and \(\left[\begin{smallmatrix}-0.7&-0.7\end{smallmatrix}\right]\) with mirrored scores. Which one is wrong?
Neither. A joint sign flip of a component with its scores preserves projections and reconstructions; only mixed or unmirrored comparisons fail. -
Two features have variances 1 and 10000. What happens without scaling, and what changes with it?
Without scaling the first component follows the large-variance feature almost entirely. Standardizing puts both on equal footing; do it when units are arbitrary. -
Randomized PCA reports slightly different ratios on each seed. Is that a bug?
No, it is an approximation. Fix the seed for reproducibility and use the exact solver when exact ratios matter.
Completion criterion
Stop when
Your blank-file PCA matches sklearn components up to sign and explained variance to 1e-6 on two datasets, and your tests still pass after negating any component.
Mark passed, then open 4.4 Only mark this when you can do it without notes.Leads to
- 4.4 k-means: objective, initialization, empty clusters and instability Cluster in PCA space and compare stability with and without reduction.
- 3.5 k-nearest neighbors, search cost and the curse of dimensionality Connect variance concentration to distance behavior in high dimensions.
Comes after
- 1.7 SVD, pseudoinverse and low-rank approximationApply maximum-variance PCA to data; do not repeat its centered 3 by 2 example here.
- 4.2 Interactions, feature selection and fit-state serializationFeed the fitted pipeline output into PCA and read reconstruction against the same split.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning Read Chapters 7 and 8 and run notebooks 07 and 08 for PCA, k-means, DBSCAN, agglomerative clustering, Gaussian mixtures and anomaly detection. Stop before extended semi-supervised and face-data projects.
- scikit-learn - decomposition, clustering and outlier detection Check decomposition orientation and solver behavior, cluster API semantics, and the distinction between novelty and outlier detection after the independent tests pass.
- ISLP - Unsupervised Learning Use Chapter 12 and its Python lab for the PCA derivations (12.2, including the low-rank approximation view and proportion of variance explained) and k-means (12.4.1), plus stability and interpretation limits, beside the notebook route.
Also in this section
- Deep-ML - unsupervised problems Use 17 K-Means Clustering, 19 PCA, 341 Gaussian Mixture Model with EM, 259 DBSCAN and 367 Isolation Forest as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including the 367 path-length convention.
- scikit-learn - preprocessing data Read 8.3.1 standardization with the sparse
with_mean=Falserule, 8.3.1.3 scaling data with outliers, and 8.3.4 categorical encoding withhandle_unknown, infrequent andsparse_outputbehavior. Apply after the HOML notebook build works. - scikit-learn - pipelines and composite estimators Read 8.1.1
Pipelinechaining with the no-leakage rule and transformer caching, and 8.1.4ColumnTransformerfor heterogeneous columns withremainderand fitted-state behavior. Use it to assemble and serialize the feature-selection pipeline. - scikit-learn - imputation of missing values Read 8.4.2 univariate median and
most_frequentstrategies, 8.4.6 Marking imputed values withMissingIndicatororadd_indicator=Truefor missingness flags, and 8.4.7 estimators that acceptNaNdirectly. Decide impute-only versus impute-plus-indicator per column. - UMAP - basic usage and parameters Work the Penguin scaling and
fit_transformsection and the Digitsfitandtransformsection, then read the parameters page for then_neighborslocal-global tradeoff andmin_distpacking. Stay in 2D exploratory views with fixed seeds.