Logistic regression and stable binary cross-entropy

1. Logistic regression: likelihood, gradient and regularization

   Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression

   Sections 3.2 (conditional log likelihood with the gradient-ascent update from zero weights at \(\eta\) 0.01) and 3.3 (\(L_2\)-penalized log likelihood, Gaussian-prior MAP reading, penalized gradient rule 26)

   Depth · Read the likelihood and gradient derivations fully. Section 3.1 writes \(P(Y = 1 \mid X) = 1/(1 + \exp(w_0 + \sum w_i X_i))\), the opposite sign convention, and 3.2 flips \(Y\) in (24)-(25) to the usual form, so check signs when you compare. Skim 3.1; its Gaussian-to-sigmoid bridge returns in the LDA unit.

   Done
2. The logistic model and maximum likelihood

   An Introduction to Statistical Learning with Python

   Chapter 4, Sections 4.3.1 to 4.3.3 (pp. 139-141): the logistic model, likelihood (4.5) preferred over nonlinear least squares, and the Default-data arithmetic (balance $2,000 gives 58.6%)

   Done
3. Maximum likelihood and gradient-based fitting

   Dive into Deep Learning - linear models slice

   Section 22.7: The Maximum Likelihood Principle, then Numerical Optimization and the Negative Log-Likelihood with its set-up, initialize, descend and check-output worked example

   Done
4. Binary logistic loss, C semantics and solvers

   scikit-learn - linear models, discriminant analysis and naive Bayes

   Section 1.1.11 with 1.1.11.1 (expit form, averaged binary cross-entropy plus \(r(w)/(S C)\); \(C = \infty\) means no penalty) and 1.1.11.3 (solver-versus-penalty table; lbfgs is the default)

   Done
5. Estimating probabilities and decision boundaries

   Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models

   Book Chapter 4: Estimating Probabilities through Decision Boundaries (printed pp. 168-174). Notebook 04 cells: Estimating Probabilities (iris virginica detector on petal width) and Decision Boundaries (boundary contour with \(C = 2\))

   Done

Focus on

• The gradient is prediction error times input, summed over rows. Mitchell ascends the summed log likelihood from zero weights; sklearn minimizes averaged cross-entropy plus an \(L_2\) term (\(C = 1.0\) by default) with lbfgs. Their optima agree only when the penalties match, for example \(C\) = np.inf against Mitchell's unpenalized rule.
• sklearn regularizes by default at \(C = 1.0\) on an averaged loss, so its \(C\) is not a textbook \(\lambda\): smaller \(C\) means stronger penalty, and \(C = \infty\) recovers the unpenalized fit.
• Never exponentiate before logging. In float64 the sigmoid rounds to exactly 1.0 once \(z\) passes about 36.7, but to exactly 0 only below about -709.8, so the naive loss for \(y = 0\) is already \(\infty\) at \(z = 37\), and at \(z = 36\) it returns 36.044 instead of 36.0. The form \(\max(z,0) - zy + \operatorname{log1p}(\exp(-|z|))\) stays exact.

Reproduce closed-book

Traps

• Equating sklearn \(C\) with a textbook \(\lambda\). \(C\) is inverse strength on an averaged loss, so doubling \(C\) halves the penalty; sample and class weights rescale the effective \(C\) through the total weight \(S\).
• Trusting probabilities from a near-unpenalized fit on nearly separable data. The likelihood keeps rewarding larger weights, so scores saturate toward exactly 0 and 1 without becoming more accurate.
• Starting gradient ascent from large random weights for symmetry breaking. The objective is concave with one global optimum and Mitchell starts at zero; large starts only waste steps or overflow the first sigmoid.

Practice

Self-test

1. Why does logistic regression need iterative fitting when linear regression has a closed form?

   Show answer Got it Missed

2. sklearn at default C = 1.0 disagrees with your zero-penalty NumPy fit. Which one is right?

   Show answer Got it Missed

3. Training loss prints inf on the first epoch for a positive example with logit near -800. What happened, and what is the fix?

   Show answer Got it Missed

4. Recompute ISLP's 58.6% default risk for balance $2,000 from Table 4.1.

   Show answer Got it Missed

Leads to

• 2.4 Softmax regression and one-versus-rest Generalize the sigmoid, the log likelihood and the stable loss to K classes.
• 2.8 Probability estimates, class priors and asymmetric thresholds Turn the fitted probabilities into cost-aware decisions.

Comes after

• 2.4 Gradients of logistic, softmax cross-entropy and regularized lossesStable BCE and the \((p - y)x\) gradient apply this derivation to logistic regression.
• 5.5 Newton and quasi-Newton methodsRecognize the curvature and stationarity checks behind iterative logistic solvers.
• 2.2 Conditioning, basis expansions, ridge, lasso and elastic netCarry penalties and standardization into penalized log likelihood.

Resources for this unit

• An Introduction to Statistical Learning with Python
  James, Witten, Hastie, Tibshirani and Taylorprimarytheory
  Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
• Dive into Deep Learning - linear models slice
  D2L authorsrequiredimplementation
  Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
• scikit-learn - linear models, discriminant analysis and naive Bayes
  scikit-learnreferencereference
  Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
• Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
  Aurelien Geron (O'Reilly, 2025)supportimplementation
  Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
• Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression
  Tom Mitchell, Carnegie Mellon Universityrequiredtheory
  Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.