Unit 1.3 · about 5 h
172 / 324Computation graphs and reverse accumulation
Frameworks hide this loop, but every gradient bug in later sections is a broken graph, a wrong local derivative or a missed accumulation. The engine you build here is the reference for all of them.
- derive
- implement
You can now
- Draw an operation graph, label every local derivative, and hand-run reverse accumulation in topological order.
- Implement a scalar autodiff engine with topological backward that matches centered finite differences.
- Explain vector-Jacobian products: the reverse pass multiplies each incoming gradient by the local Jacobian, seeded with 1 for a scalar loss.
Covers
- Computation graphs and local derivatives
- Vector-Jacobian products and reverse accumulation
Study, in this order
3 steps
- Backpropagation from toy model to algorithmUnderstanding Deep Learning - Chapters 3 to 7
Chapter 7, Section 7.2 Observations 1 and 2 (printed pp. 97 to 99) for saved forward activations and ripple effects; Section 7.3 toy example forward (Eq 7.9, p. 101) and backward (Eq 7.10 to 7.16, pp. 101 to 103); Section 7.4 with the summary (Eq 7.17 to 7.26, pp. 103 to 106)
- Backprop equations and graph costD2L - tensors, autograd, MLP and backpropagation
Forward Propagation, Computational Graph of Forward Propagation, and Backpropagation: the \(W_2\) and \(W_1\) gradient equations; Training Neural Networks: why training memory exceeds prediction memory and grows with layers times batch
Depth · The worked model omits biases by statement; Exercise 2 asks you to add one. Do not mistake the \(W\)-only equations for a complete MLP.
- The vector-Jacobian seedD2L - tensors, autograd, MLP and backpropagation
Opening paragraphs on the computational graph and backpropagation; Section 2.5.2 Backward for Non-Scalar Variables:
y.backward(gradient=torch.ones(len(y)))computing \(v^\top \partial y / \partial x\)
Focus on
- Forward pass saves what backward needs: UDL Observation 1 stores every hidden activation because weight gradients are computed from them (\(dL/dW_2\) needs \(h\)). Saving is a design obligation, not caching.
- Reverse accumulation visits nodes in reverse topological order and multiplies the incoming gradient by each local derivative. That product is the vector-Jacobian product; a scalar loss seeds it with 1.
- PyTorch
backwardon a non-scalar needs the seed vector explicitly (thegradientargument), while MXNet and TensorFlow sum first. A missing seed is an error, not an implicit sum, in PyTorch. - Training memory exceeds prediction memory because intermediates are retained for backward, scaling with layers times batch (D2L). Prediction keeps only the current layer.
Reproduce closed-book
Value objects with topological backward and match centered differences on the neuron graph.Traps
- Running backward in forward order. A node must accumulate every downstream contribution before propagating further; forward order uses unfinished gradients.
- Assuming saved activations are optional. Without the forward values, local derivatives at nonlinearities and the outer products for weights cannot be evaluated.
- Calling
backwardtwice on the same graph to get the same gradient. The second call re-accumulates (a later unit shows why); kept graphs also keep their saved memory alive.
Practice
TensorTonic: Topologically Sort a Computation DAG (autograd-l05-topological-sort)
Automatic Reverse-Mode Autodiff section: Topologically Sort a Computation DAG, then Run Reverse-Mode Autodiff
Implement both in your own repository; the sort must handle shared nodes visited once.
D2L Backprop Exercises: add a bias
Exercises: add a bias to the hidden layer, draw the computational graph, and derive the forward and backward equations
Compare your bias gradient against the affine-unit rule \(dL/db\) = row-summed upstream gradient.
Self-test
-
Why does the forward pass store every hidden activation?
Backward evaluates local derivatives at the forward values: weight gradients need \(h\) (\(dL/dW_2\) uses \(h^\top\)) and nonlinearity derivatives need their inputs. Without them the chain rule has no numbers. -
A node feeds two downstream ops. What is its reverse-mode gradient?
The sum of the vector-Jacobian products over both outgoing edges, by the multivariate chain rule. Accumulation is covered fully in the next unit. -
In PyTorch, what does
y.backward()do wheny = x * xis a vector?It raises an error. PyTorch needs an explicit seed vector, as iny.backward(gradient=torch.ones(len(y))), which computes \(v^\top \partial y / \partial x\). -
For loss \((f_3 - y)^2\), what is \(dL/df_3\)?
\(2(f_3 - y)\) (UDL Eq 7.11). It seeds the whole reverse chain. -
When is forward-mode differentiation cheaper than reverse mode?
Roughly when there are fewer inputs than outputs: forward mode costs about one pass per input dimension, reverse mode about one pass per output. A scalar loss with many parameters favors reverse mode.
Completion criterion
Stop when
Your engine differentiates the scalar neuron graph, matches centered differences, and you can hand-run the UDL toy chain and name every saved value backward needs.
Mark passed, then open 1.4 Only mark this when you can do it without notes.Leads to
- 1.4 Branches, broadcasting and saved values in backward Handle shared nodes, broadcasting and saved values.
- 1.5 Gradient checks with finite differences Verify the engine against finite differences.
Comes after
- 2.1 Limits, partial and directional derivatives, and gradientsReverse accumulation computes these partials on a graph; check it against centered differences.
- 2.3 The chain rule and matrix derivativesThe scalar engine mechanizes this chain rule with topological backward; check it against centered differences.
- 2.6 Backpropagation through dense layers and gradient checksUse these checks to validate the scalar engine against centered differences.
- 1.1 Affine maps, nonlinearities and logitsDifferentiate affine chains with the chain rule.
- 1.2 Stable softmax, cross-entropy and loss reductionsPush the loss gradient back through the network.
Resources for this unit
- Understanding Deep Learning - Chapters 3 to 7 Read Chapters 3-7, emphasizing Chapter 5 for likelihood-derived losses and Chapter 7 for backpropagation and initialization. Use notebooks 3.1, 4.1, 5.1, 6.1 and 7.1 only as equation and plot companions.
- D2L - tensors, autograd, MLP and backpropagation Use chapter_preliminaries ndarray and autograd, then MLP, backpropagation, and numerical stability and initialization. Recreate scalar operations, affine maps, activation, reduction, stable cross-entropy, topological backward and unbroadcast in a blank file.
Also in this section
- TensorTonic - Build Micrograd from Scratch Use the 25-problem Micrograd study plan in order for derivatives, gradient checks, value graphs, reverse mode, accumulation, operators, modules, losses and training. Preserve local tests as the evidence instead of a platform score.
- Deep-ML candidate - autograd and loss problems After the five-sample platform check passes, use at most one of 26 Basic Autograd Operations, 997 Manual Cross-Entropy Backprop or 1227 Stable Softmax as a changed-prompt drill.
- CS231n Linear Classification - Softmax classifier and numeric stability Read the Softmax classifier section for the cross-entropy form \(L_i = -f_{y_i} + \log \sum_j e^{f_j}\), then the Practical issues: Numeric stability part for the max-subtraction code
f -= np.max(f). - CS231n Neural Networks 3 - Gradient checks Read the Gradient Checks section only: centered formula with \(O(h^2)\) error, relative error with thresholds, double precision, step size \(h\) near 1e-5, kinks, few datapoints, burn-in, regularization and dropout discipline, and checking a few dimensions per parameter.
- PyTorch Broadcasting semantics Read General semantics for the trailing-dimension rules, prepend-1 alignment and max-size result, with the
(5, 1, 4, 1)plus(3, 1, 1)examples; read In-place semantics for why an in-place tensor may not change shape. - `torch.autograd.gradcheck` API and tolerances Read the
gradchecksignature defaults (eps1e-6,atol1e-5,rtol1e-3) and the note that inputs with overlapping memory, such asexpandoutputs, will likely fail the check. - `torch.nn.functional.cross_entropy` - `reduction` and `ignore_index` Read the
ignore_indexandreductionparameters: ignored targets contribute no gradient, and mean reduction averages over the non-ignored targets only. - PyTorch Tensor Views Read the view and storage sharing example (
data_ptr, mutating the view changes the base), the transpose non-contiguous example, the full view-ops list, and the special notes:reshapemay copy,contiguouscopies only when needed, basic indexing returns views while advanced indexing copies. - `torch.Tensor.stride` - stride and storage offset Read the stride definition: strides are per-dimension element steps, and with
storage_offsetthey locate every element. Combine with the Tensor Views page for transposed and expanded layouts. - PyTorch Autograd mechanics - in-place operations Read In-place operations with autograd (why in-place rewrites the graph and is discouraged) and In-place correctness checks (the version counter and the error on accessing a modified saved tensor).
- `torch.nn.Module` - parameters, buffers and state Read the parameters and buffers documentation: modules collect parameters recursively for the optimizer, while
register_bufferholds persistent non-parameter tensors saved alongside them.