Unit 2.5 · about 3 h
140 / 324Residual, leverage, separation and misspecification diagnostics
A fit that converges is not a fit that is right. Diagnostics separate a broken model from a broken assumption, and separation is the one failure that looks like perfect success.
- measure
You can now
- Read residual-versus-fitted plots to separate nonlinearity, heteroscedasticity and outliers, and name the repair each prescribes.
- Compute leverages from the hat matrix, flag high-leverage points, and distinguish leverage from influence.
- Detect complete separation in logistic fits from runaway coefficients and saturated probabilities, and repair it with an explicit penalty.
Covers
- Residual, leverage, separation and misspecification diagnostics
Study, in this order
5 steps
- Potential problems in the regression modelAn Introduction to Statistical Learning with Python
Chapter 3, Section 3.3.3 (pp. 100-108): nonlinearity, correlation of error terms, heteroscedasticity, outliers, high-leverage points and collinearity with the variance inflation factor
- Separation instability of logistic estimatesAn Introduction to Statistical Learning with Python
Chapter 4, Section 4.4 opening (p. 146): logistic parameter estimates turn surprisingly unstable under substantial class separation, which motivates the generative alternatives
- Penalty strength as the separation leverscikit-learn - linear models, discriminant analysis and naive Bayes
Section 1.1.11 Binary Case: the penalty enters as \(r(w)/(S C)\), so relaxing \(C\) toward \(\infty\) recovers the unpenalized fit whose weights are unbounded under separation
- Learning curves for bias-variance diagnosisHands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
Book Chapter 4: Learning Curves (printed pp. 154-159). Notebook 04 cells: Learning Curves section: sklearn
learning_curvewith RMSE on the linear fit, then the degree-10 polynomial pipeline, train versus validation error against training-set size - Error-only confusion viewsHands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
Book Chapter 3: Error Analysis (printed pp. 126-130). Notebook 03 cells: Error Analysis section: row-normalized matrices, then error-only views with
sample_weight, normalized by row and by column
Focus on
- Leverage measures position in \(x\)-space, not wrongness. The \(x = 10\) point in a 0-to-2 design carries leverage 0.98 out of \(p = 2\) while anchoring the slope; residuals measure wrongness, and influence needs both.
- Patterns have signatures. Curvature in residual-versus-fitted means nonlinearity, a funnel means heteroscedasticity, and isolated large standardized residuals mean outliers; each prescribes a different repair.
- Separation shows as coefficients that keep growing as \(C\) is relaxed, with probabilities glued to 0 and 1. On \(x = 0,1,2,3\) with \(y = 0,0,1,1\) the slope is 0.96 at \(C = 1\) and 9.31 at \(C\) = 1e3. At \(C\) = 1e6
lbfgsstops at 16.57 under its defaulttol = 1e-4with no warning, whiletol = 1e-12reaches 21.50. The fix is a finite penalty, not more iterations.
Reproduce closed-book
[[1,0],[1,1],[1,2],[1,10]] and match leverages [0.4183, 0.3307, 0.2749, 0.9761] summing to \(p = 2\). Then fit LogisticRegression on x = [0,1,2,3] with y = [0,0,1,1] at \(C = 1\), 1e3 and 1e6 with scikit-learn 1.9.1 defaults and match slopes [0.9583], [9.3057] and [16.5719]. Refit \(C\) = 1e6 with tol=1e-12 and watch the slope move to 21.4953: under separation the stopping rule, not the data, sets the weights.Traps
- Deleting every high-leverage point. Leverage is not guilt: a far-x point with a small residual anchors the fit. Act on influence, which combines residual and leverage (Cook's distance), not on position alone.
- Reading saturated 0 and 1 probabilities as a great classifier. Under separation the likelihood rewards ever-larger weights, so perfect scores signal an unidentified model rather than a solved problem.
- Raising
max_iterorCto force a separated fit to converge. The penalized optimum keeps moving outward asCgrows, and under the defaulttollbfgsstops early without aConvergenceWarning, so a quiet fit proves nothing. Keep the penalty finite or collect overlapping cases.
Practice
HoML 04 Learning Curves: bias-variance diagnosis from error curves
Learning Curves section: RMSE train and validation curves from learning_curve for the linear fit and the degree-10 polynomial pipeline
Rebuild both curve pairs from a blank script and name which regime, bias or variance dominated, each pair shows.
HoML 03 Error Analysis: error-only confusion views
Error Analysis section: row-normalized matrix, then the sample_weight error-only views normalized by row and by column
List the three most confused digit pairs from the error-only views and inspect sample images for each.
Self-test
-
Residual-versus-fitted shows a U shape. What is wrong, and what changes?
Nonlinearity the linear form cannot express. Add polynomial or interaction terms rather than collecting more data for the same line. -
The \(x = 10\) point has leverage 0.98 out of \(p = 2\). Should it be deleted?
Not on leverage alone. It dominates the slope estimate but a small residual there is informative. Check influence by refitting without it before deciding. -
Logistic coefficients keep growing as
Cis relaxed (0.96, 9.31, 16.57 atC = 1, 1e3, 1e6) and the probabilities print as 0 or 1. Diagnose and repair.Complete separation: the maximum likelihood weights are unbounded and only the penalty keeps them finite, so the penalized optimum moves outward roughly in proportion to \(\log C\). Keep \(C\) finite, confirm the weights hold under a tightertol, and look for overlapping cases to collect. -
Standardized residuals fan out as fitted values grow. Name the violation and one valid response.
Heteroscedasticity, non-constant error variance. Transform the response, model the variance, or use variance-robust inference; adding polynomial terms does not fix it.
Completion criterion
Stop when
You can name the violated assumption from any residual plot, reproduce the leverage table from a blank file, and catch separation from a \(C\) sweep before looking at accuracy.
Mark passed, then open 2.6 Only mark this when you can do it without notes.Leads to
- 2.8 Probability estimates, class priors and asymmetric thresholds Set the decision rule only after the fit survives diagnostics.
- 5.7 Learning curves, error taxonomies and slices Fold residual and slice analysis into the evaluation harness.
Comes after
- 2.1 Least squares: residuals, normal equations and stable solversRead residuals and leverage on the fits built here.
- 2.2 Conditioning, basis expansions, ridge, lasso and elastic netDiagnose the collinearity and scale symptoms that penalties only mask.
- 2.4 Softmax regression and one-versus-restWatch separation inflate the weights here exactly as in binary logistic regression.
Resources for this unit
- An Introduction to Statistical Learning with Python Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
- scikit-learn - linear models, discriminant analysis and naive Bayes Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
- Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
Also in this section
- Dive into Deep Learning - linear models slice Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
- NeetCode ML - linear-model closing drill After the section gate, solve Gradient Descent, Sigmoid and ReLU, Softmax, Cross-Entropy Loss, Linear Regression Forward and Linear Regression Training cold. Repeat only failed items from a blank file with changed values.
- Deep-ML - linear and probabilistic model problems Use 15 Linear Regression Using Gradient Descent, 106 Train Logistic Regression with Gradient Descent and 140 Bernoulli Naive Bayes Classifier as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin. The local implementation remains the evidence.
- Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.