Unit 2.6 · about 4 h
141 / 324LDA, QDA and generative versus discriminative classifiers
LDA, QDA, naive Bayes and logistic regression all draw linear-or-quadratic boundaries from different assumptions. Seeing which assumption buys which boundary tells you what breaks first when the data disobeys.
- derive
You can now
- Derive the LDA discriminant from Gaussian class conditionals with shared covariance and show the boundary is linear.
- Derive the QDA and Gaussian naive Bayes discriminants, and state the exact assumption that turns one into the other.
- Contrast generative and discriminative fitting on parameter counts, small-sample behavior and robustness to violated assumptions.
Covers
- LDA and QDA equations, assumptions and decision boundaries
- Generative versus discriminative classification
Study, in this order
5 steps
- LDA for one and many predictorsAn Introduction to Statistical Learning with Python
Chapter 4, Sections 4.4.1 to 4.4.2 (pp. 147-155): Bayes theorem for classification, Gaussian class conditionals, the shared-covariance assumption and the linear discriminant
- QDA and the analytical comparison of classifiersAn Introduction to Statistical Learning with Python
Chapter 4, Section 4.4.3 (p. 156) for QDA, then Section 4.5.1 (pp. 161-164): the linear, quadratic and additive log-odds forms and which method nests inside which
- Generative versus discriminative classifiersTom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression
Sections 1 (Bayes-rule learning and the \(2(2n - 1)\) parameter count), 2.1-2.2 (conditional independence and the naive Bayes derivation), 3.1 (Gaussian naive Bayes implies the logistic form) and 4 (discriminative versus generative, asymptotic identity, Ng and Jordan rates)
- LDA and QDA formulation and solversscikit-learn - linear models, discriminant analysis and naive Bayes
Section 1.2.2 (Bayes rule, Gaussian class conditionals, QDA log-posterior, LDA linear form with
coef_andintercept_, and the QDA-with-diagonal-covariance equalsGaussianNBnote) and 1.2.5 (svdis the default solver) - LDA and QDA lab on the Smarket dataAn Introduction to Statistical Learning with Python
Linear Discriminant Analysis and Quadratic Discriminant Analysis sections: fits, posterior values and confusion matrices beside the logistic fit
Focus on
- LDA log odds are linear in \(x\) because the shared covariance cancels every quadratic term; QDA keeps one covariance per class, so its boundary is quadratic with cross terms.
- Gaussian naive Bayes is QDA with diagonal class covariances. If each feature's variance is also shared across classes, it becomes LDA with a diagonal shared covariance (ISLP 4.5.1). Naive Bayes in general, with any per-feature densities, and QDA do not nest: naive Bayes log odds are additive in the features, while QDA's contain products of feature pairs.
- Logistic regression and LDA posit the same linear log-odds form but fit it differently: LDA estimates class Gaussians and priors while logistic maximizes the conditional likelihood. Expect LDA to win when normality holds and logistic when it does not.
Reproduce closed-book
default_rng(3): 40 standard normal points around [0,0], then 40 around [2,2]. Match sklearn LDA priors [0.5, 0.5], training score 0.95, coefficients [1.927, 2.229] and intercept -4.070.Traps
- Fitting QDA with too few rows per class. Each class covariance comes from that class's rows alone, and with p or fewer rows per class (the class mean uses one) it is singular. Use QDA's
reg_paramorshrinkagewith theeigensolver, or LDA withshrinkage. - Calling LDA parameter-free. It exposes no tuning knob by default but still estimates K means, a shared covariance and priors, which overfit small samples.
- Assuming discriminative always wins with enough data. Logistic regression often has the lower asymptotic error under misspecification, but per Ng and Jordan naive Bayes approaches its own asymptote in order \(\log n\) examples versus order \(n\) for logistic regression, where \(n\) is the number of features. So LDA and naive Bayes often win when training data are scarce.
Practice
ISLP Lab 4.7.3-4.7.4: LDA and QDA on the Smarket data
Linear Discriminant Analysis and Quadratic Discriminant Analysis sections: fits, posteriors and confusion matrices against the Section 4.7.2 logistic fit
Record where QDA beats LDA and where it loses, and tie each case to the shared-versus-separate covariance assumption.
Self-test
-
Write the two-class LDA decision rule and name each estimated quantity.
Assign \(\operatorname{argmax}\) over \(k\) of \(x^\top\Sigma^{-1}\mu_k - \mu_k^\top\Sigma^{-1}\mu_k/2 + \log\pi_k\): class means, one shared covariance and priors. The \(x\)-quadratic terms cancel because \(\Sigma\) is shared, leaving a linear boundary. -
When does QDA collapse to Gaussian naive Bayes, and when does Gaussian naive Bayes collapse toward LDA?
QDA restricted to diagonal class covariances is Gaussian naive Bayes. Gaussian naive Bayes whose per-feature variances are shared across classes is LDA with a diagonal shared covariance. General naive Bayes and QDA do not nest: one is additive, the other has pairwise products. -
50 training points, 20 features, roughly Gaussian classes. LDA, QDA or logistic first?
LDA, possibly with shrinkage. QDA must estimate two 20-by-20 covariances from about 25 points each, and logistic fits 21 weights discriminatively with higher variance. Revisit logistic if normality is badly violated and more data arrives. -
State the Ng and Jordan sample-complexity contrast and its practical consequence.
Naive Bayes approaches its asymptotic error in order \(\log n\) training examples and logistic regression needs order \(n\), where \(n\) is the number of features, not samples. So naive Bayes often wins on small training sets, while logistic regression, with its usually lower asymptotic error, wins once data are plentiful.
Completion criterion
Stop when
You can write both discriminants from the Gaussian log posterior, predict the boundary shape from the covariance assumption, and call the likely winner from data geometry before fitting.
Mark passed, then open 2.7 Only mark this when you can do it without notes.Leads to
- 2.7 Naive Bayes: event models, smoothing and log-space scoring Shrink the Gaussian assumption to diagonal covariance and smoothed counts.
- 2.8 Probability estimates, class priors and asymmetric thresholds Weight discriminant scores by class priors before deciding.
Comes after
- 3.4 Common distribution families and transformationsLDA and QDA fit Gaussian class-conditional densities.
Resources for this unit
- An Introduction to Statistical Learning with Python Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
- scikit-learn - linear models, discriminant analysis and naive Bayes Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
- Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.
Also in this section
- Dive into Deep Learning - linear models slice Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
- Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
- NeetCode ML - linear-model closing drill After the section gate, solve Gradient Descent, Sigmoid and ReLU, Softmax, Cross-Entropy Loss, Linear Regression Forward and Linear Regression Training cold. Repeat only failed items from a blank file with changed values.
- Deep-ML - linear and probabilistic model problems Use 15 Linear Regression Using Gradient Descent, 106 Train Logistic Regression with Gradient Descent and 140 Bernoulli Naive Bayes Classifier as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin. The local implementation remains the evidence.