Unit 2.6 · about 3 h
64 / 324Backpropagation through dense layers and gradient checks
Backprop hides a sign, transpose or accumulation bug in every new layer. A two-evaluation directional check finds it in seconds; a per-coordinate loop costs \(2mn\) evaluations.
- derive
- implement
You can now
- Derive backprop through an affine map, an elementwise nonlinearity and a branched sum, with shapes stated.
- Implement centered-difference gradient checks including random-direction tests for matrix parameters, with tolerance and norm stated.
- Diagnose step-size, cancellation, nondifferentiability and stochastic-state failures from the error curve.
Covers
- Backpropagation through dense layers and nonlinearities
- Finite-difference gradient checking and its failure modes
Study, in this order
4 steps
- Finite-difference approximations and their errorsMIT 18.S096 - Matrix Calculus for ML
Lecture 3, Part 2 notes (course notes Chapter 4): forward differences and first-order truncation error (Sections 4.2-4.5), roundoff and the step-size rule of thumb (Section 4.6), centered differences and Richardson extrapolation (Section 4.7)
- Numeric differentiation and the reverse engineHands-On ML - Autodiff notebook
Cells Numeric differentiation (
gradients()takes one-sided differences witheps = 1e-4and \(n + 1\) calls; nesting it gives a Hessian whose exact 0 entry prints as -1.4e-6) and Reverse mode autodiff (hand engine withbackpropagate, the artifact to test against differences) - Backprop equations with parent accumulationMathematics for Machine Learning - Ch. 5
Section 5.6 (pp. 160-164): backward computation working from the output with accumulation over parent nodes (5.135)-(5.145); the dense-layer backward pattern
- Reverse mode on computational graphsMIT 18.S096 - Matrix Calculus for ML
Lecture 5, Part 3 notes (course notes Chapter 8, Section 8.3): computational graphs, summing products along every input-to-output path, and reverse mode on graphs (Section 8.3.1)
Focus on
- Centered differences carry \(O(h^2)\) error until rounding at about \(\epsilon/h\) takes over; \(h = 10^{-5}\) in
float64sits near the bottom of that U-curve. - The random-direction test \(\langle G, V \rangle\) against the directional difference needs 2 evaluations for any shape and catches transpose and scale bugs. Use several seeded \(V\).
- Backprop through a branch sums cotangents from every downstream use. A missing accumulation is the classic silent bug.
- Relative error needs a scale: use \(\|\text{num} - \text{ana}\| / (\|\text{num}\| + \|\text{ana}\|)\) so near-zero gradients do not explode the ratio.
Reproduce closed-book
standard_normal from numpy.random.default_rng(5) in float64: \(\delta = (a - t) \odot (1 - a^2)\), \(dL/dW = \delta x^\top\). Measured: \(L\) = 1.73395370, centered differences (\(h = 10^{-5}\)) give relative error \(\|\text{num} - \text{ana}\|/(\|\text{num}\| + \|\text{ana}\|)\) = 3.4e-11, and the direction test \(|\langle dW, V \rangle - (L(W + hV) - L(W - hV))/(2h)|\) = 1.4e-10. Repeat at \(h = 0.1\) to get relative error 4.3e-3, consistent with \(O(h^2)\) truncation.Traps
- Checking at one \(h\) far from \(10^{-5}\) without a sweep. For the
tanhlayer in Reproduce, \(h = 0.1\) leaves 4.3e-3 relative error and hides real bugs above the truncation floor, while \(h = 10^{-12}\) drowns in cancellation. - Gradchecking through a kink or a stochastic op and reading the mismatch as a backprop bug. Freeze stochasticity and keep nondifferentiable points out of the check.
- Comparing with
==or to too many digits. State the norm and tolerance; for thetanhlayer,float64centered differences bottom out near 1e-10, not 1e-16.
Practice
Deep-ML 313: Numerical Gradient Checking
Implement numerical_gradient_check(f, x, analytical_grad, epsilon) with centered differences; tests check quadratics against rel_err < 1e-5
The signature defaults epsilon to 1e-7. Sweep \(h\), explain why 1e-5 is safer in float64, then keep your dense-layer check from Reproduce as the oracle the section mini-project extends with a seeded gradient error.
Self-test
-
Write the backward pass for \(z = Wx + b\), \(a = \tanh(z)\), \(L = 0.5\|a - t\|^2\), with shapes.
\(\delta = (a - t) \odot (1 - a^2)\), shape \(m\). \(dL/dW = \delta x^\top\) (\(m \times n\)), \(dL/db = \delta\), \(dL/dx = W^\top \delta\). The outer product, not an inner product, gives \(m \times n\). -
Why is centered \(O(h^2)\) while one-sided is \(O(h)\)?
Taylor expansion: the forward difference leaves a first-order term \(f''h/2\), while centering cancels all odd powers of \(h\), leaving \(O(h^2)\). The remainder form needs \(C^3\). -
\(W\) has 6 entries. Compare per-coordinate and directional check cost.
Per-coordinate costs 12 evaluations; one direction costs 2. Run 3 to 5 random directions from a seeded generator and record the seed,dtypeand norm. -
The check disagrees only at entries whose pre-activation sits near zero under ReLU. Is backprop wrong?
Probably not. Centered differences average left and right slopes at a nondifferentiable kink. Move the check point or use a smooth activation for the check.
Completion criterion
Stop when
Your dense-layer backward matches centered differences to 1e-8 relative and a directional test to 1e-9 absolute, and you can move the error up and down the U-curve by changing \(h\).
Mark passed, then open 3.1 Only mark this when you can do it without notes.Leads to
- 1.5 Gradient checks with finite differences That unit standardizes centered differences in
float64with \(h = 10^{-5}\) and relative error; this unit's directional test is its matrix-valued companion. - 1.3 Computation graphs and reverse accumulation Use these checks to validate the scalar engine against centered differences.
Comes after
- 2.5 Forward mode, reverse mode, JVPs, VJPs and Hessian-vector productsDense-layer backprop is reverse mode at scale, and gradient checks test it.
Resources for this unit
- MIT 18.S096 - Matrix Calculus for ML Use lectures 1-5, the Hessian portion of lecture 7 and computational-graph portion of lecture 8; do the supplied problems and notebooks.
- Hands-On ML - Autodiff notebook Work through numerical differentiation and both AD modes, then rebuild the essential reverse-mode engine independently.
- Mathematics for Machine Learning - Ch. 5 Use Chapter 5 as the compact derivative, Jacobian, chain-rule and backpropagation reference.
Also in this section
- PyTorch Autograd mechanics Consult after the handwritten engine for accumulation, graph behavior, disabled tracking and vector-Jacobian products.
- MIT 18.02SC - Multivariable Calculus Unit 2 Use sessions 24-38 only if partial derivatives, gradients, Hessians or the multivariable chain rule are weak.
- CS231n - Linear Classification Read the Softmax classifier section for the cross-entropy loss formula with both the information-theory and MLE views, and Practical issues: Numeric stability for the max-subtraction trick. Derive the softmax gradient by hand; do not implement the course assignment here.