Softmax regression and one-versus-rest

1. Softmax, vectorization caution and cross-entropy loss

   Dive into Deep Learning - linear models slice

   Section 4.1: The Softmax (ordering preservation, argmax needs no exponentials), Vectorization (row-wise softmax with the overflow-underflow caution), Loss Function with Log-Likelihood and Softmax and Cross-Entropy Loss

   Done
2. Softmax from scratch, including the instability warning

   Dive into Deep Learning - linear models slice

   Section 4.4: The Softmax (naive implementation with the do-not-use-verbatim caution), The Model, The Cross-Entropy Loss, Training, Prediction

   Depth · Read the naive softmax as the failure demo. Use the framework softmax with built-in protections for every real fit.

   Done
3. Multinomial logistic regression

   An Introduction to Statistical Learning with Python

   Chapter 4, Section 4.3.5 (p. 144): the \(K\)-class extension (4.13) with log odds linear in \(x\) (4.14)

   Done
4. Multiclass logistic form and gradient

   Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression

   Section 3.4 (\(K - 1\) linear functions in equations 27-28, regularized gradient rule 29)

   Done
5. Multinomial geometry, OvR solvers and calibration

   scikit-learn - linear models, discriminant analysis and naive Bayes

   Section 1.1.11.2 (deliberate \(K\)-vector overparameterization, non-unique unpenalized solutions) and 1.1.11.3 with its solver table and Solvers' details (liblinear cannot fit a multinomial model, so multiclass use wraps it in OneVsRestClassifier; the multinomial loss is expected to calibrate better than one-versus-rest)

   Done
6. One-versus-rest in practice

   Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models

   Book Chapter 3: Multiclass Classification (printed pp. 124-126). Notebook 03 cells: Multiclass Classification section: the SVC OvO-training note, OneVsRestClassifier with its 10 estimators, and the SGDClassifier multiclass run

   Done

Focus on

• Softmax preserves order, so \(\operatorname{argmax}\) needs no exponentials. Probabilities still need the max-subtracted form, because \(\exp(1000)\) overflows to \(\infty\) and \(\infty\) divided by \(\infty\) is nan.
• sklearn fits \(K\) vectors for \(K\) classes on purpose, for symmetry under regularization. Without penalty that redundancy makes the solution non-unique, which is one more reason the default \(L_2\) matters.
• One-versus-rest trains \(K\) binary problems and compares uncalibrated margins, while multinomial optimizes one joint loss. Expect worse-calibrated probabilities from the OvR scheme.

Reproduce closed-book

Traps

• Copying the D2L 4.4 teaching softmax into real code. The page itself warns against verbatim use; framework implementations subtract the row maximum internally.
• Calling OvR margins probabilities. One-versus-rest decision_function scores are not calibrated across the K classifiers; take predict_proba from a joint fit or calibrate afterwards.
• Passing multi_class to LogisticRegression under sklearn 1.9. The parameter is gone and raises TypeError. Multiclass fits are multinomial under the default lbfgs, liblinear raises a ValueError on three or more classes, and one-versus-rest now means wrapping the estimator in OneVsRestClassifier.

Practice

Self-test

1. Why can argmax skip the softmax entirely?

   Show answer Got it Missed

2. sklearn fits 3 vectors for 3 classes while Mitchell fits \(K - 1 = 2\) linear functions. Is one of them wrong?

   Show answer Got it Missed

3. Naive softmax of [1000, 1001, 999] returns nan. Show the exact failure and the fix.

   Show answer Got it Missed

4. liblinear is your only allowed solver on a 5-class problem. Which scheme, and what do you lose?

   Show answer Got it Missed

Leads to

• 2.8 Probability estimates, class priors and asymmetric thresholds Turn multiclass scores into decisions with costs attached.
• 2.5 Residual, leverage, separation and misspecification diagnostics Watch separation inflate the weights here exactly as in binary logistic regression.

Comes after

• 2.3 Logistic regression and stable binary cross-entropyGeneralize the sigmoid, the log likelihood and the stable loss to K classes.

Resources for this unit

• An Introduction to Statistical Learning with Python
  James, Witten, Hastie, Tibshirani and Taylorprimarytheory
  Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
• Dive into Deep Learning - linear models slice
  D2L authorsrequiredimplementation
  Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
• scikit-learn - linear models, discriminant analysis and naive Bayes
  scikit-learnreferencereference
  Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
• Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
  Aurelien Geron (O'Reilly, 2025)supportimplementation
  Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
• Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression
  Tom Mitchell, Carnegie Mellon Universityrequiredtheory
  Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.