Section 2 of 5 · Classical Machine Learning

Linear & Probabilistic Prediction

Linear and probabilistic models expose their assumptions. They connect likelihood, optimization, regularization, probabilities and decision thresholds before flexible models make those choices harder to see.

8 units 18 topics 0/8

Units

8 in this section

2.1 Least squares: residuals, normal equations and stable solvers
derive · implement
4 h
2.2 Conditioning, basis expansions, ridge, lasso and elastic net
derive · measure
5 h
2.3 Logistic regression and stable binary cross-entropy
derive · implement
4 h
2.4 Softmax regression and one-versus-rest
derive · implement
3 h
2.5 Residual, leverage, separation and misspecification diagnostics
measure
3 h
2.6 LDA, QDA and generative versus discriminative classifiers
derive
4 h
2.7 Naive Bayes: event models, smoothing and log-space scoring
derive · implement
3 h
2.8 Probability estimates, class priors and asymmetric thresholds
derive · measure
2 h

Before moving on

Checkpoint

Derive and independently implement linear regression, logistic regression and one naive Bayes variant, then diagnose one numerical failure and one assumption failure on a changed dataset.

Deliberately out of scope

Leave for later

Full generalized linear model theory
Bayesian posterior computation
Specialized robust and high-dimensional estimators

Resources for this section

Shared by every unit above

An Introduction to Statistical Learning with Python
James, Witten, Hastie, Tibshirani and Taylortheory

primary

Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
Dive into Deep Learning - linear models slice
D2L authorsimplementation

required

Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
scikit-learn - linear models, discriminant analysis and naive Bayes
scikit-learnreference

reference

Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
Aurelien Geron (O'Reilly, 2025)implementation

support

Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
NeetCode ML - linear-model closing drill
NeetCodepractice

support

After the section gate, solve Gradient Descent, Sigmoid and ReLU, Softmax, Cross-Entropy Loss, Linear Regression Forward and Linear Regression Training cold. Repeat only failed items from a blank file with changed values.
Deep-ML - linear and probabilistic model problems
Deep-MLpractice

support

Use 15 Linear Regression Using Gradient Descent, 106 Train Logistic Regression with Gradient Descent and 140 Bernoulli Naive Bayes Classifier as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin. The local implementation remains the evidence.
Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression
Tom Mitchell, Carnegie Mellon Universitytheory

required

Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.

Apply the section

Mini-project · Linear and probabilistic models from scratch, with two failure diagnoses (10 to 14 h)

Implement least squares, ridge, logistic regression and one naive Bayes variant in NumPy in your own repository, prove each against scikit-learn and finite differences, then diagnose one numerical failure and one assumption failure on changed datasets you construct.

Steps

Implement least squares three ways: normal equations, an SVD-based solve and minibatch gradient descent with a decaying step. Pass a float64 gradient check and match LinearRegression, handling the intercept the same way on both sides.
Add ridge with the centered closed form and a penalty path. Match Ridge to 1e-12 and report validation error across \(\alpha\) on a fixed split.
Implement logistic regression on logits with the stable loss \(\max(z,0) - zy + \operatorname{log1p}(\exp(-|z|))\) and an \(L_2\) penalty. Pass a gradient check, then match LogisticRegression on its own objective (\(C\) times the summed loss plus half the squared norm) with tol tightened.
Implement Bernoulli or multinomial naive Bayes with Laplace smoothing in log space. Match scikit-learn's feature_log_prob_ to 1e-8 and handle unseen values and single-class training data.
Numerical failure: build a changed dataset that breaks one naive path, such as a near-collinear design, extreme logits for a naive loss, or a separable set. Show the failure, the measurement that discriminates its cause, the repair and a regression test.
Assumption failure: build a dataset that violates one model assumption, such as XOR-labeled binary features for naive Bayes, heteroscedastic noise for least squares, or non-Gaussian classes for LDA. Show the diagnostic that reveals it and the response, validated on held-out data.
Turn one fitted logistic model into a decision: derive the cost-optimal threshold for stated costs, apply it on validation, and move it for a stated prevalence change without refitting.

Deliverables

The NumPy package: least squares (three solvers), ridge, logistic regression and naive Bayes.
The test suite: scikit-learn differential tests, finite-difference checks and the extreme-input probes.
Derivation notes: normal equations, the ridge closed form, the logistic gradient and the smoothed MAP rule.
Two failure reports, numerical and assumption, each with diagnostic, repair and regression test.

Checks

external-tests · Deep-ML 15, 106 and 140
rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Correctness against oracles	Each implementation matches its scikit-learn counterpart within a stated tolerance.	Property tests vary seeds, shapes and penalties, and every analytic gradient matches central differences in `float64`.
Numerical stability	The logit loss stays finite on extreme logit probes, and ill-conditioned designs use an SVD-based solve.	The notes show where each naive form breaks and why the stable form stays exact.
Numerical failure diagnosis	One failure is reproduced, explained, repaired and guarded by a test.	The discriminating measurement rules out a rival cause, for example a tighter `tol` showing that `lbfgs` stopped early rather than converged.
Assumption failure	One violated assumption is shown with a diagnostic and a response.	The response is validated on held-out data and the report states what it still cannot fix.
Decision	The threshold follows from stated costs and is chosen on validation data.	The prevalence adjustment is derived from prior odds and checked against a refit on resampled data.

Feeds into: Milestone artifact 2, the mechanisms suite: the regularized linear or logistic model and naive Bayes come from here. The failure reports feed the decision dossier.