Dimensionality Reduction & PCA

PCA finds directions of high variance

Principal component analysis rotates a centered numeric feature space into orthogonal directions ordered by the variance they explain. The first component captures the largest possible variance; the next captures the largest remaining variance subject to orthogonality. Component scores are coordinates in this new basis, and loadings describe how original features contribute. PCA is not feature selection: it combines variables into new axes.

from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline

pca_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=0.95)),
])
X_train_reduced = pca_pipeline.fit_transform(X_train_numeric)

Fit the projection only on training rows

PCA learns directions from data, so fitting it before the train/test split leaks information about the held-out feature distribution. Put scaling and PCA inside a pipeline so each training fold learns its own projection. explained_variance_ratio_ tells how much variance each component captures; a cumulative plot helps choose how many components retain a desired fraction. A 2D projection is useful for exploration, but it can hide class structure and does not prove visible clusters are real.