Unit 4.7 · about 3 h
158 / 324Outlier and novelty detection, thresholds and representation
Anomaly scores spend budgets: each alert costs review time. A threshold without a policy and a score without a representation note cannot be operated.
- explain
- measure
You can now
- Separate outlier detection on dirty training data from novelty detection on clean training data with new points.
- Set thresholds from contamination policy or score percentiles and report the implied alert rate.
- Show that every anomaly score depends on representation: scaling, PCA width and kernel change the ranking.
Covers
- Outlier versus novelty detection and threshold policy
- Representation dependence of anomaly scores
Study, in this order
2 steps
- Density thresholds and representation-dependent anomaliesHands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning
Book Chapter 8: Using Gaussian Mixtures for Anomaly Detection (printed pp. 274-275) and Exercises 12 and 13 (p. 281). Notebook 08 cells: Anomaly Detection Using Gaussian Mixtures: the 2 percent density rule via percentile on
score_samples; Exercises 12 and 13: GMM on PCA-reduced Olivetti faces with sampling and rotated, flipped and darkened anomaly checks, plus the PCA reconstruction-error detector - Novelty versus outlier contract and detectorsscikit-learn - decomposition, clustering and outlier detection
2.7 Novelty and Outlier Detection: outlier versus novelty definitions,
contamination,offset_,decision_functionsign andpredictlabels; 2.7.2 novelty prediction on new data; 2.7.3EllipticEnveloperobust covariance,IsolationForestpath lengths and LOF local density with itsfit_predict-only default; 2.7.4 novelty detection with LOF (novelty=True)
Focus on
- Outlier methods fit on polluted data; novelty methods fit on clean data and judge new points. LOF with
novelty=Falsehas nopredictfor new data; refit withnovelty=True. contaminationsets the score cutoff as a percentile throughoffset_. It declares the alert rate; it does not discover it.IsolationForestisolates with random feature splits; anomalies need fewer splits. In the paper's score, near 1 means anomaly, well below 0.5 means normal, and all scores near 0.5 mean no distinct anomaly; scikit-learn'sscore_samplesreturns the negative. Trees stop at depth \(\lceil\log_2(n)\rceil\) for \(n\) samples per tree.- Rescale or re-encode and the ranking changes: LOF neighborhoods, elliptic envelopes and GMM densities all live in the given coordinates.
Reproduce closed-book
IsolationForest with contamination=0.25 and random_state=42 on \(\left[\begin{smallmatrix}0\\0.1\\0.2\\10.0\end{smallmatrix}\right]\): predict gives [1,1,1,-1] with only the last decision_function negative. State the implied alert rate and why the cutoff is a policy.Traps
- Tuning
contaminationuntil the known incident appears, then reporting the hit as detection. The threshold was fit to the answer; hold it fixed or cross-check on other windows. - Calling LOF
fit_predicton a deployment batch. Withnovelty=Falsethe batch is judged against itself, not against clean history, andpredictdoes not exist. Fit on clean data withnovelty=True, then callpredicton new points. - Comparing raw
score_samplesacross models or scalings. Scores are model-and-coordinate specific; compare ranks or calibrated alert rates.
Practice
Deep-ML 367: Implement Isolation Forest for Anomaly Detection
Problem 367: implement isolation_forest(X, n_trees, sample_size, random_state) with random feature and uniform-split trees, depth cap ceil(log2(sample_size)) and BST-normalized scores
The tests check rounded score vectors on a 5-point set with one outlier at random_state=42 and a tight cluster at random_state=123. They match only the learn section's recipe taken literally: np.random.seed, legacy np.random calls, and \(c(n) = 2(\ln(n-1) + 0.5772156649) - 2(n-1)/n\) for every leaf size above 1, so \(c(2)\) is about 0.154. The paper and scikit-learn set \(c(2) = 1\), which fails these vectors; note that difference rather than treating it as a bug.
Self-test
-
Training data is clean and the task judges next week points. Outlier or novelty setup?
Novelty: fit on clean history and predict new points. An outlier setup would fit the polluted batch including the points under test. -
contamination=0.25on 100 training points. How many train alerts, and what doesoffset_do?About 25:offset_is the score percentile implementing that rate. It does not estimate the true anomaly fraction. -
Standardizing a skewed sensor channel shuffles LOF rankings. Is that a bug?
No. LOF compares local densities in the given units; rescaling rewrites every neighborhood. Record scaling as part of the detector. -
A GMM 2 percent rule flags a new batch at 9 percent. What do you report?
The flag rate rose from 2 to 9 percent, 4.5 times, under a fixed threshold. Report the shift with counts and slices before touching the threshold.
Completion criterion
Stop when
You match both Deep-ML score vectors, reproduce the [1,1,1,-1] contamination demo, and can flip a detector between outlier and novelty setups with the right predict calls.
Leads to
- 5.5 Cost-sensitive thresholds, imbalance and prevalence change Carry the threshold policy into cost-sensitive decisions under shifted prevalence.
- 5.9 Covariate, label and concept shift, drift and label delay Reuse novelty scores as drift signals with detection-delay notes.
Comes after
- 4.5 Gaussian mixtures and EMSpend mixture densities as anomaly scores with a declared threshold.
- 4.6 DBSCAN, agglomerative clustering and cluster validationTreat DBSCAN noise labels as candidates, not verdicts, with a scored threshold.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - dimensionality reduction and unsupervised learning Read Chapters 7 and 8 and run notebooks 07 and 08 for PCA, k-means, DBSCAN, agglomerative clustering, Gaussian mixtures and anomaly detection. Stop before extended semi-supervised and face-data projects.
- scikit-learn - decomposition, clustering and outlier detection Check decomposition orientation and solver behavior, cluster API semantics, and the distinction between novelty and outlier detection after the independent tests pass.
Also in this section
- ISLP - Unsupervised Learning Use Chapter 12 and its Python lab for the PCA derivations (12.2, including the low-rank approximation view and proportion of variance explained) and k-means (12.4.1), plus stability and interpretation limits, beside the notebook route.
- Deep-ML - unsupervised problems Use 17 K-Means Clustering, 19 PCA, 341 Gaussian Mixture Model with EM, 259 DBSCAN and 367 Isolation Forest as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including the 367 path-length convention.
- scikit-learn - preprocessing data Read 8.3.1 standardization with the sparse
with_mean=Falserule, 8.3.1.3 scaling data with outliers, and 8.3.4 categorical encoding withhandle_unknown, infrequent andsparse_outputbehavior. Apply after the HOML notebook build works. - scikit-learn - pipelines and composite estimators Read 8.1.1
Pipelinechaining with the no-leakage rule and transformer caching, and 8.1.4ColumnTransformerfor heterogeneous columns withremainderand fitted-state behavior. Use it to assemble and serialize the feature-selection pipeline. - scikit-learn - imputation of missing values Read 8.4.2 univariate median and
most_frequentstrategies, 8.4.6 Marking imputed values withMissingIndicatororadd_indicator=Truefor missingness flags, and 8.4.7 estimators that acceptNaNdirectly. Decide impute-only versus impute-plus-indicator per column. - UMAP - basic usage and parameters Work the Penguin scaling and
fit_transformsection and the Digitsfitandtransformsection, then read the parameters page for then_neighborslocal-global tradeoff andmin_distpacking. Stay in 2D exploratory views with fixed seeds.