Unit 2.2 · about 5 h
137 / 324Conditioning, basis expansions, ridge, lasso and elastic net
An unpenalized fit trusts every column equally, so one rescaled feature or one duplicated signal can swing the coefficients wildly. Bases and penalties decide what the model may express before more data can save it.
- derive
- measure
You can now
- Derive the ridge closed form and predict how standardization changes the penalized solution.
- Contrast \(L_1\) and \(L_2\) penalties on correlated features: selection and sparsity versus shrinkage and stability.
- Measure coefficient paths and validation error across the penalty strength on fixed splits and select it from evidence.
Covers
- Conditioning, feature scale and coefficient instability
- Polynomial and interaction bases
- Ridge, lasso and elastic net regularization
Study, in this order
6 steps
- Polynomial fitting and model complexityDive into Deep Learning - linear models slice
Section 3.6.1.1 Model Complexity, then 3.6.2.1 Polynomial Curve Fitting (with distinct \(x\) values, a polynomial whose degree equals the number of examples fits the training set exactly) and Fig. 3.6.1, the underfitting-overfitting curve
- Weight decay and the high-dimensional demoDive into Deep Learning - linear models slice
Section 3.7: Norms and Weight Decay, then High-Dimensional Linear Regression with \(d = 200\) features on 20 training examples at noise std 0.01
- Polynomial and interaction bases, then ridge, lasso and tuningAn Introduction to Statistical Learning with Python
Chapter 3, Section 3.3.2 (pp. 94-99) for polynomial and interaction extensions; Chapter 6, Sections 6.2.1 to 6.2.3 (pp. 240-252) for ridge, the lasso and tuning-parameter selection
- Regularization lab on the Hitters dataAn Introduction to Statistical Learning with Python
Ridge Regression and the Lasso sections: solution paths on Hitters Salary with 5-fold cross-validation of the tuning parameter
- Ridge, lasso, elastic net and polynomial objectivesscikit-learn - linear models, discriminant analysis and naive Bayes
Sections 1.1.2 (
Ridgeminimizes RSS plus \(\alpha\) times squared norm), 1.1.3 (Lassominimizes \(\frac{1}{2n}\) RSS plus \(\alpha\) times L1 norm, coordinate descent), 1.1.5 (Elastic-Net objective and the correlated-features guidance) and 1.1.16 (PolynomialFeaturesbasis expansion) - Polynomial regression and regularized linear modelsHands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
Book Chapter 4: Polynomial Regression (printed pp. 153-154) and Regularized Linear Models (pp. 159-167). Notebook 04 cells: Polynomial Regression, Ridge Regression, Lasso Regression, Elastic Net and Early Stopping sections, including the scale-and-regularization warning in the chapter Q and A
Focus on
Ridgeadds \(\alpha\|w\|^2\) to the raw RSS, whileLassoandElasticNetadd their \(\alpha\) penalty to \(\text{RSS}/(2n)\), so \(\alpha = 1\) means very different strengths. Compare paths or CV errors, never raw alphas.- Penalties tax raw coefficient size. A feature with small numeric values, such as kilometers instead of meters, needs a larger weight for the same effect, so it pays more penalty and is shrunk harder. Standardize before penalizing and leave the intercept free.
Lassolands on corners: the sklearn guide says it is likely to pick one of several correlated features at random, while elastic net is likely to keep both. D2L's \(d = 200\), \(n = 20\) demo shows weight decay rescuing a fit with more features than samples.
Reproduce closed-book
default_rng(2), \(n = 100\), \(x_1\) and \(x_2\) standard normal, \(x_3 = x_1 + 0.1\,\text{noise}\), \(y = 2x_1 - 3x_2\) plus standard normal noise, drawn in that order. Show Ridge(alpha=1.0) matches the centered closed form to 1e-14 at [1.33909931, -3.06183881, 0.64520852]. Then show Lasso returns all zeros once \(\alpha\) reaches \(\max|X_c^\top y_c|/n\) = 2.74, the \(x_2\) entry; just below it only \(x_2\) enters (-0.146 at \(\alpha = 2.6\)). Ridge at any finite \(\alpha\) keeps every coefficient nonzero here.Traps
- Judging
Ridge(alpha=1)againstLasso(alpha=1)as a fair fight. The objectives normalize the loss differently, so match effective shrinkage or validation error instead. - Fitting the scaler on all the data, or penalizing the intercept. Fit standardization on train only and keep the offset out of the penalty.
- Reading lasso zeros as proof the dropped features carry no signal. Across resamples a correlated rival is selected instead, so report selection stability rather than one mask.
Practice
D2L 3.7 Exercise 1: \(\lambda\) sweep on the high-dimensional demo
Exercises, problem 1: vary \(\lambda\) and plot training and validation error; then problem 3 for the \(L_1\) update equations
Keep the \(d = 200\), \(n = 20\) demo fixed, record where validation error bottoms out, and state the update change the L1 penalty forces.
ISLP Lab 6.5.2: Ridge Regression and the Lasso
Ridge Regression and the Lasso sections on the Hitters Salary data with 5-fold CV selection of the tuning parameter
Compare the selected strengths and explain any disagreement on which coefficients survive.
Self-test
-
Ridge(alpha=1)andLasso(alpha=1)disagree sharply on standardized data. Is one implementation wrong?No.Ridgeadds \(\alpha\|w\|^2\) to the raw RSS, whileLassoadds \(\alpha\|w\|_1\) to \(\text{RSS}/(2n)\), so the same number applies very different pressure. Sweep a path for each and compare by validation error. -
D2L 3.7 fits \(d = 200\) features on 20 examples. Why does plain least squares break, and what does the \(L_2\) term change algebraically?
With 200 features and 20 rows, \(X^\top X\) has rank at most 20, so infinitely many weight vectors fit training exactly. Adding \(\alpha I\) makes \(X^\top X + \alpha I\) invertible, so the solution is unique, and it trades some training fit for a smaller norm; as \(\alpha\) goes to 0 it tends to the minimum-norm interpolant. -
Two features correlate at 0.999.
Lassokeeps one with a large weight while ridge keeps both, shrunken. Why?When \(x_1\) and \(x_2\) are nearly equal, the loss depends almost only on \(w_1 + w_2\). For a fixed sum with both weights positive, the \(L_1\) norm is the same for every split, so the lasso is nearly indifferent and small differences in the data tip it to a corner with one weight at zero. The \(L_2\) norm is smallest at an equal split, so ridge shares the weight. -
With 10 distinct \(x\) values, a degree-9 polynomial fits the training data exactly. Is it the best model?
No. Zero training error does not imply low validation error: the exact fit can swing wildly between and beyond the points. That gap is what the underfitting-overfitting curve diagnoses.
Completion criterion
Stop when
Your ridge closed form matches sklearn to 1e-12, your \(\lambda\) sweep shows validation error falling then rising, and you can explain any lasso-versus-ridge selection disagreement on correlated features.
Mark passed, then open 2.3 Only mark this when you can do it without notes.Leads to
- 2.5 Residual, leverage, separation and misspecification diagnostics Diagnose the collinearity and scale symptoms that penalties only mask.
- 2.3 Logistic regression and stable binary cross-entropy Carry penalties and standardization into penalized log likelihood.
Comes after
- 1.2 Linear maps, span, independence and change of basisApply the distinction between the map and its coordinate representation.
- 1.6 Eigenvalues, symmetric and positive-semidefinite matricesUse PSD geometry to understand ridge's shifted Gram spectrum.
- 4.2 Maximum likelihood and MAP estimationConnect Gaussian-prior MAP to ridge regression.
- 5.1 Convexity, optimality conditions and conditioningApply curvature and conditioning to ridge regression.
- 5.4 Constraints, KKT conditions and proximal methodsContrast ridge's smooth penalty with lasso's nonsmooth sparsity mechanism.
- 2.1 Least squares: residuals, normal equations and stable solversRepair the singular and badly scaled designs met here with standardization and penalties.
- 4.2 Interactions, feature selection and fit-state serializationConnect scaling and basis choices to optimizer geometry and penalties.
Resources for this unit
- An Introduction to Statistical Learning with Python Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
- Dive into Deep Learning - linear models slice Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
- scikit-learn - linear models, discriminant analysis and naive Bayes Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
- Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
Also in this section
- NeetCode ML - linear-model closing drill After the section gate, solve Gradient Descent, Sigmoid and ReLU, Softmax, Cross-Entropy Loss, Linear Regression Forward and Linear Regression Training cold. Repeat only failed items from a blank file with changed values.
- Deep-ML - linear and probabilistic model problems Use 15 Linear Regression Using Gradient Descent, 106 Train Logistic Regression with Gradient Descent and 140 Bernoulli Naive Bayes Classifier as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin. The local implementation remains the evidence.
- Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.