Unit 4.5 · about 3 h
156 / 324Gaussian mixtures and EM
Mixtures turn distance into probability: overlapping groups get graded membership plus density scores that the anomaly unit spends.
- derive
You can now
- Explain responsibilities as posterior component membership and the EM loop as alternating \(E\) and \(M\) steps.
- Derive the 1D M-step updates for means, variances and weights from weighted maximum likelihood.
- Select the component count with BIC or AIC and state why a plain
GaussianMixturealways uses every component.
Covers
- Gaussian mixtures, responsibilities and EM intuition
Study, in this order
2 steps
- Mixture fits, covariances and component countHands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning
Book Chapter 8: Gaussian Mixtures through Selecting the Number of Clusters (printed pp. 269-278). Notebook 08 cells: Gaussian Mixtures: EM-fitted weights, means and covariances on the three-ellipsoid data with the
covariance_typefull,tied,sphericalanddiagcomparison; Selecting the Number of Clusters with the BIC and AIC formulas and the minimum at \(k=3\)Depth · Skim Bayesian Gaussian Mixture Models; variational inference is later work. Keep the BIC and AIC minimum plot.
- EM estimation, init and failure modesscikit-learn - decomposition, clustering and outlier detection
2.1.1 Gaussian Mixture: EM E-step and M-step description, covariance options, the singularity and always-uses-all-components cons, and the BIC asymptotic note; Choice of the Initialization method with the k-means default
Focus on
- Responsibilities are posteriors that sum to one per point. The M-step refits each Gaussian to responsibility-weighted data.
- Likelihood rises every EM round but only to a local optimum. Starts matter; the sklearn default starts from k-means.
- Full covariance needs \(d(d+1)/2\) parameters per component. Few points per component invite singular covariances;
reg_covar(1e-6 by default) adds to the diagonal to keep them invertible. - BIC penalizes parameters with \(\log(m) \times p\). It recovers the true count only asymptotically under a true mixture; on real data it is a guide, not proof.
Reproduce closed-book
Traps
- Dividing the M-step variance by \(n-1\). Mixture MLE divides by the responsibility sum, which is \(n\) for \(K=1\), giving 2.6667 on \(\left[\begin{smallmatrix}2&4&6\end{smallmatrix}\right]\), not 4.0.
- Reading converged likelihood as a global optimum. Different starts converge differently; report spread, not one run.
- Trusting the BIC minimum as truth on misspecified data. The docs note asymptotic recovery needs data truly drawn from a mixture.
Practice
Deep-ML 341: Gaussian Mixture Model with EM Algorithm
Problem 341: implement fit_gmm_1d with \(E\) and \(M\) steps for n_iterations, rounding means, variances and weights to 4 decimals
Test one checks the \(K=1\) closed form with means \(\left[\begin{smallmatrix}4.0\end{smallmatrix}\right]\), variances \(\left[\begin{smallmatrix}2.6667\end{smallmatrix}\right]\) and weights \(\left[\begin{smallmatrix}1.0\end{smallmatrix}\right]\); test two checks a symmetric \(K=2\) case stays at means \(\left[\begin{smallmatrix}0.5&9.5\end{smallmatrix}\right]\) with variances \(\left[\begin{smallmatrix}0.25&0.25\end{smallmatrix}\right]\). Derive both by hand first.
Self-test
-
What does responsibility 0.9 of point \(x\) under component 2 mean?
Posterior membership: under current parameters the probability is 0.9 that component 2 generated \(x\). It is not a hard label. -
\(K=1\) EM on \(\left[\begin{smallmatrix}2&4&6\end{smallmatrix}\right]\) gives variance 2.6667. Derive it.
Mean 4.0; squared deviations 4+0+4=8; MLE divides by 3. The unbiased 4.0 answers a different question. -
Weights converge to \(\left[\begin{smallmatrix}0.99&0.01\end{smallmatrix}\right]\) on two overlapping groups. What do you conclude?
One component explains nearly everything. Try \(K=1\), check BIC, and consider that the small component fits outliers. -
Full covariance on 50 points in 10 dimensions with \(K=5\). What breaks?
Each full covariance needs 55 parameters from about 10 effective points. Expect singularities; usetiedordiag, regularize, or cut \(K\).
Completion criterion
Stop when
You derive the \(K=1\) numbers by hand, pass both Deep-ML tests, and can predict how tied versus full covariance changes the parameter count.
Mark passed, then open 4.6 Only mark this when you can do it without notes.Leads to
- 4.7 Outlier and novelty detection, thresholds and representation Spend mixture densities as anomaly scores with a declared threshold.
- 4.6 DBSCAN, agglomerative clustering and cluster validation Compare soft parametric groups with density and merge structure.
Comes after
- 3.3 Expectation, variance, covariance and conditioning lawsEM iterates a conditional expectation of the complete-data log likelihood.
- 4.4 k-means: objective, initialization, empty clusters and instabilitySoften hard assignments into responsibilities.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning Read Chapters 7 and 8 and run notebooks 07 and 08 for PCA, k-means, DBSCAN, agglomerative clustering, Gaussian mixtures and anomaly detection. Stop before extended semi-supervised and face-data projects.
- scikit-learn - decomposition, clustering and outlier detection Check decomposition orientation and solver behavior, cluster API semantics, and the distinction between novelty and outlier detection after the independent tests pass.
Also in this section
- ISLP - Unsupervised Learning Use Chapter 12 and its Python lab for the PCA derivations (12.2, including the low-rank approximation view and proportion of variance explained) and k-means (12.4.1), plus stability and interpretation limits, beside the notebook route.
- Deep-ML - unsupervised problems Use 17 K-Means Clustering, 19 PCA, 341 Gaussian Mixture Model with EM, 259 DBSCAN and 367 Isolation Forest as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including the 367 path-length convention.
- scikit-learn - preprocessing data Read 8.3.1 standardization with the sparse
with_mean=Falserule, 8.3.1.3 scaling data with outliers, and 8.3.4 categorical encoding withhandle_unknown, infrequent andsparse_outputbehavior. Apply after the HOML notebook build works. - scikit-learn - pipelines and composite estimators Read 8.1.1
Pipelinechaining with the no-leakage rule and transformer caching, and 8.1.4ColumnTransformerfor heterogeneous columns withremainderand fitted-state behavior. Use it to assemble and serialize the feature-selection pipeline. - scikit-learn - imputation of missing values Read 8.4.2 univariate median and
most_frequentstrategies, 8.4.6 Marking imputed values withMissingIndicatororadd_indicator=Truefor missingness flags, and 8.4.7 estimators that acceptNaNdirectly. Decide impute-only versus impute-plus-indicator per column. - UMAP - basic usage and parameters Work the Penguin scaling and
fit_transformsection and the Digitsfitandtransformsection, then read the parameters page for then_neighborslocal-global tradeoff andmin_distpacking. Stay in 2D exploratory views with fixed seeds.