Section 1 of 8 · Deep Learning
Neural Computation, Losses & Reverse Mode
Deep learning becomes debuggable when tensor semantics, objective reductions and reverse accumulation are understood below the framework surface.
7 units 17 topics 0/7
Units
7 in this section
- 1.1 Affine maps, nonlinearities and logits explain · derive4 h
- 1.2 Stable softmax, cross-entropy and loss reductions derive · implement5 h
- 1.3 Computation graphs and reverse accumulation derive · implement5 h
- 1.4 Branches, broadcasting and saved values in backward derive · implement5 h
- 1.5 Gradient checks with finite differences implement · measure3 h
- 1.6 Tensor layout: shape, stride, views and copies explain · implement3 h
- 1.7 Aliasing, mutation and module ownership explain · measure3 h
Before moving on
Checkpoint
From a blank file, implement reverse accumulation for a branched broadcast graph, match finite differences and PyTorch within justified tolerance, and explain each saved value and mutation hazard.
Deliberately out of scope
Leave for later
- Higher-order autodiff and symbolic differentiation
- Framework compiler internals
- Large neural architectures
Resources for this section
Shared by every unit above
- Understanding Deep Learning - Chapters 3 to 7 Read Chapters 3-7, emphasizing Chapter 5 for likelihood-derived losses and Chapter 7 for backpropagation and initialization. Use notebooks 3.1, 4.1, 5.1, 6.1 and 7.1 only as equation and plot companions.
- D2L - tensors, autograd, MLP and backpropagation Use chapter_preliminaries ndarray and autograd, then MLP, backpropagation, and numerical stability and initialization. Recreate scalar operations, affine maps, activation, reduction, stable cross-entropy, topological backward and unbroadcast in a blank file.
- TensorTonic - Build Micrograd from Scratch Use the 25-problem Micrograd study plan in order for derivatives, gradient checks, value graphs, reverse mode, accumulation, operators, modules, losses and training. Preserve local tests as the evidence instead of a platform score.
- Deep-ML candidate - autograd and loss problems After the five-sample platform check passes, use at most one of 26 Basic Autograd Operations, 997 Manual Cross-Entropy Backprop or 1227 Stable Softmax as a changed-prompt drill.
- CS231n Linear Classification - Softmax classifier and numeric stability Read the Softmax classifier section for the cross-entropy form \(L_i = -f_{y_i} + \log \sum_j e^{f_j}\), then the Practical issues: Numeric stability part for the max-subtraction code
f -= np.max(f). - CS231n Neural Networks 3 - Gradient checks Read the Gradient Checks section only: centered formula with \(O(h^2)\) error, relative error with thresholds, double precision, step size \(h\) near 1e-5, kinks, few datapoints, burn-in, regularization and dropout discipline, and checking a few dimensions per parameter.
- PyTorch Broadcasting semantics Read General semantics for the trailing-dimension rules, prepend-1 alignment and max-size result, with the
(5, 1, 4, 1)plus(3, 1, 1)examples; read In-place semantics for why an in-place tensor may not change shape. - `torch.autograd.gradcheck` API and tolerances Read the
gradchecksignature defaults (eps1e-6,atol1e-5,rtol1e-3) and the note that inputs with overlapping memory, such asexpandoutputs, will likely fail the check. - `torch.nn.functional.cross_entropy` - `reduction` and `ignore_index` Read the
ignore_indexandreductionparameters: ignored targets contribute no gradient, and mean reduction averages over the non-ignored targets only. - PyTorch Tensor Views Read the view and storage sharing example (
data_ptr, mutating the view changes the base), the transpose non-contiguous example, the full view-ops list, and the special notes:reshapemay copy,contiguouscopies only when needed, basic indexing returns views while advanced indexing copies. - `torch.Tensor.stride` - stride and storage offset Read the stride definition: strides are per-dimension element steps, and with
storage_offsetthey locate every element. Combine with the Tensor Views page for transposed and expanded layouts. - PyTorch Autograd mechanics - in-place operations Read In-place operations with autograd (why in-place rewrites the graph and is discouraged) and In-place correctness checks (the version counter and the error on accessing a modified saved tensor).
- `torch.nn.Module` - parameters, buffers and state Read the parameters and buffers documentation: modules collect parameters recursively for the optimizer, while
register_bufferholds persistent non-parameter tensors saved alongside them.
Apply the section
Mini-project · Tensor reverse-mode engine with differential checks (10 to 14 h)
Build a small tensor autodiff engine on NumPy in your own repository: broadcast-aware operations, reverse accumulation over branched graphs, a stable cross-entropy and one tensor MLP. Prove its gradients against finite differences and PyTorch, not against a training curve.
Steps
- Implement a
Tensorclass on NumPy withadd,mul,matmul,sumover axes,exp,log,tanhandReLU, each with abackwardrule, and abackward()that visits a topological order and accumulates into.grad. - Make every binary op broadcast-aware: sum each incoming gradient over the broadcast axes back to the operand shape. Test
(3, 1)with(1, 4),(4,)with(3, 4), and a scalar, against PyTorch. - Add log-softmax and mean cross-entropy as one fused op whose gradient is softmax minus one-hot over the batch size. Test
float32logits of magnitude 1000 for finite loss and gradients, againsttorch.nn.functional.cross_entropy. - Build a branched graph where one tensor reaches the loss through at least three paths including a broadcast. Check every leaf gradient against central differences and PyTorch in
float64, keeping the comparison as evidence. - Train a two-layer tensor MLP with your engine on a small synthetic classification set, and show its first-step gradients match a PyTorch copy with the same weights.
- Layout tests: feed transposed and noncontiguous arrays into your engine and match gradients from contiguous copies. For PyTorch views, assert shared storage through
untyped_storage().data_ptr(), and compute the element offset from strides andstorage_offsetfor aslice, atransposeand anexpand. - Reproduce one saved-value or mutation defect, such as an input edited in place between forward and backward. Show the wrong or refused gradient, fix it with a copy on save or a version counter, and keep a regression test.
Deliverables
- The engine package: ops with broadcast-aware backward, topological backward, fused cross-entropy and the MLP.
- The test suite:
float64finite-difference and PyTorch checks, broadcast cases, extreme-logit cases, layout and storage tests, and the defect regression test. - A one-page note listing the values each op saves for backward, why, and the tolerance chosen for each check.
Checks
- external-tests · TensorTonic autograd-from-scratch problems named in the units
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Gradient correctness | Every op matches PyTorch in float64 within a stated tolerance on random inputs. | The branched broadcast graph matches both finite differences and PyTorch, and the note justifies the step size and tolerance. |
| Broadcasting | Gradients reduce to each operand shape in the tested broadcast cases. | A property test draws random broadcast-compatible shape pairs and compares against PyTorch. |
| Numerical stability | Cross-entropy stays finite on extreme float32 logits. | A test shows the naive form failing on the same input, and the note explains why the max shift leaves the gradient unchanged. |
| Saved values and mutation | One defect is reproduced, fixed and guarded by a test. | The note lists every saved value per op, and the engine either refuses or survives an in-place edit of a saved input, with a test for each case. |
| Layout | Transposed and noncontiguous inputs give the same gradients as contiguous copies. | Storage and offset assertions cover a slice, a transpose and an expand, checked against PyTorch strides. |
Feeds into: Capstone artifact 1, the neural core proof pack.