Unit 2.4 · about 3 h
62 / 324Gradients of logistic, softmax cross-entropy and regularized losses
These three gradients are the output layer of almost every classifier you will train. Derive them once and later frameworks only change the packaging.
- derive
You can now
- Derive the logistic sigmoid derivative, the binary-cross-entropy-with-logits gradient \((p - y)x\), and the softmax cross-entropy gradient \(p\) minus one-hot.
- Implement the three gradients in NumPy and confirm each against centered differences with a stated tolerance.
- Explain why max-subtraction leaves softmax probabilities unchanged and where the naive formula overflows.
Covers
- Gradients of logistic, softmax cross-entropy and regularized losses
Study, in this order
2 steps
- Sigmoid, quadratic and regularizer gradientsMathematics for Machine Learning - Ch. 5
Exercise 5.2 setup (p. 170): derivative of the logistic sigmoid \(1/(1 + \exp(-x))\). Identities (5.103)-(5.108) (p. 158): \(a^\top X^{-1} b\), \(a^\top Xb\), \(x^\top Bx\) and the weighted least-squares gradient (5.108). Least-squares linear-model pattern (5.75)-(5.78) (p. 154)
- Softmax classifier and numeric stabilityCS231n - Linear Classification
Softmax classifier section: cross-entropy loss \(L_i = -f_{y_i} + \log \sum_j e^{f_j}\) with the information-theory and MLE views. Practical issues: Numeric stability: max-subtraction trick with the
[123, 456, 789]overflow exampleDepth · Read the loss formula and the stability trick only. Derive the gradient yourself in Reproduce; skip the SVM sections and the interactive demo.
Focus on
- \(\operatorname{sigmoid}'(x) = \operatorname{sigmoid}(x)(1 - \operatorname{sigmoid}(x))\) falls out of \(\exp' = \exp\); the \((p - y)\) factor comes from \(dL/dp\) cancelling \(dp/ds\).
- The softmax cross-entropy gradient is p minus one-hot because the logsumexp derivative is the softmax itself.
- \(L_2\) adds \(\lambda w\) under the \((\lambda/2)\|w\|^2\) convention and \(2 \lambda w\) under \(\lambda\|w\|^2\). State the convention before comparing numbers.
- Max-subtraction is exact in real arithmetic since it multiplies top and bottom by one constant; it only changes float rounding, which the check measures.
Reproduce closed-book
float64 NumPy: \(z = [2.0, 1.0, 0.1]\) with \(y = 0\) gives \(p = [0.659001, 0.242433, 0.098566]\), loss 0.417030, grad [-0.340999, 0.242433, 0.098566]; max-subtracted softmax agrees with naive to 2.8e-17. A single logistic example (\(w = [1, 2]\), \(x = [0.5, -1]\), \(b = 0.5\), \(y = 1\)) gives \(p = 0.268941\) and \(dL/dw = [-0.365529, 0.731059]\).Traps
- Averaging versus summing the batch loss changes the gradient by \(1/m\). A mismatched convention looks like a wrong derivation.
- Feeding probabilities into a logsumexp. Softmax cross-entropy needs logits; probabilities double-apply the exponential.
- Reading softmax outputs as calibrated probabilities. The CS231n notes show that a stronger regularization shrinks the weights and makes the same example's probabilities more diffuse, so only their ordering is meaningful.
Practice
MML Chapter 5, Exercise 5.2 (logistic sigmoid derivative)
Exercises (p. 170), Exercise 5.2: derivative of \(f(x) = 1/(1 + \exp(-x))\)
Derive the softmax cross-entropy gradient beside it and check all three gradients with the Reproduce script.
Self-test
-
Derive the logistic sigmoid derivative. Which rule and fact?
\(\operatorname{sigmoid}(x) = (1 + e^{-x})^{-1}\). Chain rule with \(d/dx e^{-x} = -e^{-x}\) gives \(\operatorname{sigmoid}(x)(1 - \operatorname{sigmoid}(x))\). -
One logistic example with \(s = w^\top x + b\) and \(L = -[y \log p + (1 - y) \log(1 - p)]\): show \(dL/dw = (p - y)x\).
\(dL/dp = (p - y)/(p(1 - p))\) and \(dp/ds = p(1 - p)\), so the product is \(p - y\); times \(ds/dw = x\). The cancellation needs \(p\) strictly inside (0, 1). -
Derive \(dL/dz\) for softmax cross-entropy with true class \(y\).
\(L = -z_y + \operatorname{logsumexp}(z)\) and \(d/dz_j \operatorname{logsumexp}(z) = p_j\), so the gradient is \(p - e_y\). -
The loss uses \((\lambda/2)\|w\|^2\) in one source and \(\lambda\|w\|^2\) in another. What changes in the gradient?
\(\lambda w\) versus \(2 \lambda w\). The minimizers agree only after rescaling \(\lambda\), so report the convention with every number.
Completion criterion
Stop when
Your hand-derived sigmoid, binary-cross-entropy and softmax gradients all match centered differences to 1e-9 relative, and you can state the L2 convention from memory.
Mark passed, then open 2.5 Only mark this when you can do it without notes.Leads to
- 1.2 Stable softmax, cross-entropy and loss reductions That unit fuses
log_softmaxwith max subtraction and uses softmax minus one-hot; this unit derived both. - 2.3 Logistic regression and stable binary cross-entropy Stable BCE and the \((p - y)x\) gradient apply this derivation to logistic regression.
Comes after
- 2.3 The chain rule and matrix derivativesLoss gradients apply these differentials to least squares, logistic and softmax objectives.
Resources for this unit
- Mathematics for Machine Learning - Ch. 5 Use Chapter 5 as the compact derivative, Jacobian, chain-rule and backpropagation reference.
- CS231n - Linear Classification Read the Softmax classifier section for the cross-entropy loss formula with both the information-theory and MLE views, and Practical issues: Numeric stability for the max-subtraction trick. Derive the softmax gradient by hand; do not implement the course assignment here.
Also in this section
- MIT 18.S096 - Matrix Calculus for ML Use lectures 1-5, the Hessian portion of lecture 7 and computational-graph portion of lecture 8; do the supplied problems and notebooks.
- Hands-On ML - Autodiff notebook Work through numerical differentiation and both AD modes, then rebuild the essential reverse-mode engine independently.
- PyTorch Autograd mechanics Consult after the handwritten engine for accumulation, graph behavior, disabled tracking and vector-Jacobian products.
- MIT 18.02SC - Multivariable Calculus Unit 2 Use sessions 24-38 only if partial derivatives, gradients, Hessians or the multivariable chain rule are weak.