Section 2 of 5 · Mathematical Foundations

Calculus & Automatic Differentiation

Training is repeated differentiation through composed programs. Shape-aware derivatives and numerical checks are the fastest way to separate a wrong model from a wrong implementation.

6 units 14 topics 0/6

Units

6 in this section

2.1 Limits, partial and directional derivatives, and gradients
derive
3 h
2.2 Jacobians, Hessians and Taylor approximations
derive
3 h
2.3 The chain rule and matrix derivatives
derive
4 h
2.4 Gradients of logistic, softmax cross-entropy and regularized losses
derive
3 h
2.5 Forward mode, reverse mode, JVPs, VJPs and Hessian-vector products
derive · implement
4 h
2.6 Backpropagation through dense layers and gradient checks
derive · implement
3 h

Before moving on

Checkpoint

Derive a branched matrix computation, implement a small reverse-mode autodiff engine, and isolate a seeded gradient error.

Deliberately out of scope

Leave for later

General differential geometry
Full symbolic differentiation systems
Higher-order implicit differentiation

Resources for this section

Shared by every unit above

MIT 18.S096 - Matrix Calculus for ML
MIT OpenCourseWarecourse

primary

Use lectures 1-5, the Hessian portion of lecture 7 and computational-graph portion of lecture 8; do the supplied problems and notebooks.
Hands-On ML - Autodiff notebook
Aurelien Geronimplementation

required

Work through numerical differentiation and both AD modes, then rebuild the essential reverse-mode engine independently.
PyTorch Autograd mechanics
PyTorch documentationreference

required

Consult after the handwritten engine for accumulation, graph behavior, disabled tracking and vector-Jacobian products.
Mathematics for Machine Learning - Ch. 5
Deisenroth, Faisal & Ongtheory

reference

Use Chapter 5 as the compact derivative, Jacobian, chain-rule and backpropagation reference.
MIT 18.02SC - Multivariable Calculus Unit 2
MIT OpenCourseWarerefresher

optional

Use sessions 24-38 only if partial derivatives, gradients, Hessians or the multivariable chain rule are weak.
CS231n - Linear Classification
Stanford Universityreference

support

Read the Softmax classifier section for the cross-entropy loss formula with both the information-theory and MLE views, and Practical issues: Numeric stability for the max-subtraction trick. Derive the softmax gradient by hand; do not implement the course assignment here.

Apply the section

Mini-project · A small reverse-mode engine and a matrix-calculus gradient-check harness (8 to 12 h)

Build a scalar reverse-mode engine with forward-mode duals and a harness that checks closed-form matrix gradients by directional finite differences. Derive one branched computation by hand, plant a gradient bug, and show the harness isolating it.

Steps

Write a scalar Value class with arithmetic, powers, exp, log and tanh, each with its local derivative, and a backward() over topological order accumulating into grad. Test one graph where a node reaches the output through several paths against a hand derivative and central differences.
Add forward mode with dual numbers. For one function from \(\mathbb{R}^3 \to \mathbb{R}^2\), check Jacobian-vector against vector-Jacobian products through \(u^\top(Jv) = (J^\top u)^\top v\) with directions of your choice.
Build a check(f, grad, X, V, h) harness comparing the directional derivative against central differences. Derive and check the trace, log-determinant, softmax cross-entropy, \(\tanh\) layer and inverse differential in float64 with directions of your choice, taking \(X\) symmetric positive definite where needed.
Sweep the step over powers of ten for two checks and keep the raw error table. Predict where the curve bends, then measure: explain the falling and rising arms and choose your harness step and tolerance from the curve.
Derive by hand the gradient of a branched computation reusing one matrix, writing each local vector-Jacobian product and its accumulation. Implement it in NumPy and check it with the harness, plus a Hessian-vector product by centered differences against the exact Hessian of a quadratic.
Plant one gradient bug, such as a missing transpose, an overwrite instead of an accumulation, or a wrong derivative sign. Show the directional check failing, localize the fault by checking each local vector-Jacobian product alone, and keep the bug as a test that must fail.

Deliverables

numlab/autodiff/: scalar reverse-mode engine, forward-mode duals and gradient-check harness.
Tests for shared-path accumulation, the JVP-VJP identity, the branched derivation, the Hessian-vector product and the planted bug.
notes/calculus.md with the branched derivation and the step-size curve with its chosen step.

Checks

rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Engine correctness	Every operation matches central differences within a stated tolerance, including a node reached through several paths.	Forward and reverse mode agree through the dot-product identity on directions of your choice.
Matrix gradients	Each closed form passes the directional check in `float64`.	Each closed form is derived in the notes from differentials, naming the assumption it needs.
Step size	The sweep table is kept as raw output and the chosen step sits near its minimum.	The note explains the truncation and cancellation arms and derives the tolerance from them.
Fault isolation	The planted bug fails the directional check and stays in the suite as a failing case.	Local products are checked one at a time to name the faulty operation.

Feeds into: Capstone reverse-mode differentiation: the engine, the harness and the planted-bug regression feeding the later tensor engine.