Unit 2.2 · about 3 h
60 / 324Jacobians, Hessians and Taylor approximations
Jacobians say how outputs move; Hessians say how gradients move. Taylor turns both into the local models that justify gradient steps and Newton steps.
- derive
You can now
- Assemble the Jacobian of a vector-valued map with consistent shapes under a stated layout.
- Write first- and second-order Taylor approximations and bound the remainder from the Hessian size.
- Classify a stationary point from Hessian eigenvalues on a small hand-worked example.
Covers
- Jacobians, Hessians and consistent shape conventions
- First- and second-order Taylor approximations
Study, in this order
3 steps
- Jacobians, Hessians and multivariate TaylorMathematics for Machine Learning - Ch. 5
Section 5.3 (pp. 149-154): vector-valued gradients, the Jacobian, the determinant-as-magnifier example, gradient of \(f(x) = Ax\) in (5.67)-(5.68). Section 5.7 (pp. 164-165): higher-order derivatives and Hessian symmetry for twice continuously differentiable functions. Section 5.8 (pp. 165-169): multivariate Taylor series (5.149)-(5.156) with the worked cubic \(f(x, y) = x^2 + 2xy + y^3\) at (1, 2): gradient (5.165), Hessian (5.171) and third derivatives (5.174)-(5.176)
- Second derivatives, bilinear forms and HessiansMIT 18.S096 - Matrix Calculus for ML
Lecture 7, Part 2 notes (course notes Chapter 12): second derivatives as symmetric bilinear forms, Hessian matrices, minima versus maxima versus saddles
- Second-derivative test in two variablesMIT 18.02SC - Multivariable Calculus Unit 2
Session 30 (Part A): second-derivative test for two-variable functions
Depth · Use only as the 2-D classification check; skip the Lagrange sessions.
Focus on
- The Jacobian of \(f: \mathbb{R}^n \to \mathbb{R}^m\) is \(m \times n\) in numerator layout, and each row is one scalar gradient.
- Hessian symmetry needs continuous second partials. Without \(C^2\), mixed partials can disagree and the Taylor form breaks.
- The first-order remainder scales with the squared step times the Hessian size; the second-order remainder with the cubed step.
- Eigenvalue signs classify: all positive is a strict local minimum, mixed signs a saddle, a zero eigenvalue leaves the test silent.
Reproduce closed-book
numpy.linalg.eigvalsh, and classify the origin as a strict local minimum since \(H\) is positive definite. Verify the second-order Taylor model is exact: \(f(0.3, -0.2) = 0.07\), and \(0.5 d^\top H d\) in float64 gives 0.06999999999999999, one ulp (1.4e-17) below.Traps
- Calling a stationary point with positive diagonal Hessian entries a minimum. Off-diagonals matter; only definiteness decides.
- Trusting a Taylor model far from the base point. The remainder grows with powers of the step, so recompute rather than extrapolate.
- Switching gradient layout mid-derivation. The layout fixes the Jacobian and Hessian shapes, so a silent switch transposes everything.
Practice
Implement hessian(f, x) as the centered-difference Jacobian of the provided gradient, and classify_critical_point(H) from eigvalsh signs with a tolerance
Its tests call diag(1e-9, 2) a saddle, but a near-zero eigenvalue leaves the second-derivative test silent: \(x^4 + y^2\) has Hessian diag(0, 2) at its minimum. Pass the tests, then return a fourth label for that case in your own copy.
MIT 18.S096 Problem Set 1, Problem 0 (\(2 \times 2\) Jacobian matrices)
Problem 0, parts (a) to (d): Jacobian matrices of the rotate, hyperbolic rotate, nonlinear shear and warp maps. Official solutions exist; attempt before reading them
Do the hyperbolic rotation by hand; its Jacobian determinant returns in Problem Set 2, Problem 4(a), which calls it Problem 1b.
Self-test
-
For \(f: \mathbb{R}^3 \to \mathbb{R}^2\), what is the Jacobian shape and what is its first row?
\(2 \times 3\) in numerator layout. The first row is the (row) gradient of \(f_1\). -
Where is the \(C^2\) assumption used in the claim that the Hessian is symmetric?
Continuity of the mixed second partials lets the two differentiation orders agree. The proof step equating the two mixed partials uses exactly this continuity. -
Approximate \(e^{0.1}\) with \(T_1\) about 0. Give the value, the remainder and a bound.
\(T_1 = 1.1\), actual value 1.10517092, remainder 0.00517092. With \(M = e^{0.1}\) bounding the second derivative on [0, 0.1], \(M h^2/2 = 0.00552585\), and the remainder satisfies the bound. -
Classify the stationary point with Hessian \(\left[\begin{smallmatrix}2&1\\1&2\end{smallmatrix}\right]\).
Eigenvalues 1 and 3, both positive, so the Hessian is positive definite and the point is a strict local minimum.
Completion criterion
Stop when
You write the Jacobian and second-order Taylor of a new \(\mathbb{R}^2\) map by hand, classify its stationary point, and NumPy eigvalsh plus a remainder evaluation agree.
Leads to
- 5.1 Convexity, optimality conditions and conditioning Hessian definiteness becomes the second-order optimality condition.
- 2.5 Forward mode, reverse mode, JVPs, VJPs and Hessian-vector products The Hessian returns as Hessian-vector products computed without forming the matrix.
Comes after
- 2.1 Limits, partial and directional derivatives, and gradientsGradients stack into Jacobian rows and Hessian entries.
Resources for this unit
- MIT 18.S096 - Matrix Calculus for ML Use lectures 1-5, the Hessian portion of lecture 7 and computational-graph portion of lecture 8; do the supplied problems and notebooks.
- Mathematics for Machine Learning - Ch. 5 Use Chapter 5 as the compact derivative, Jacobian, chain-rule and backpropagation reference.
- MIT 18.02SC - Multivariable Calculus Unit 2 Use sessions 24-38 only if partial derivatives, gradients, Hessians or the multivariable chain rule are weak.
Also in this section
- Hands-On ML - Autodiff notebook Work through numerical differentiation and both AD modes, then rebuild the essential reverse-mode engine independently.
- PyTorch Autograd mechanics Consult after the handwritten engine for accumulation, graph behavior, disabled tracking and vector-Jacobian products.
- CS231n - Linear Classification Read the Softmax classifier section for the cross-entropy loss formula with both the information-theory and MLE views, and Practical issues: Numeric stability for the max-subtraction trick. Derive the softmax gradient by hand; do not implement the course assignment here.