Unit 1.7 · about 3 h
176 / 324Aliasing, mutation and module ownership
Views alias memory, in-place ops rewrite history, and detach cuts the graph without warning. Ownership rules decide which tensor the optimizer may touch and which state merely travels along.
- explain
- measure
You can now
- Predict when an in-place or view mutation corrupts
backward, and route around it with out-of-place ops or no-grad updates. - Explain
detachas a provenance cut, and diagnose detached-path and forgotten-zeroing gradient bugs from their signatures. - State module ownership at mechanism level: recursive parameter collection, parameters versus buffers, and who may mutate what.
Covers
- Aliasing, mutation and detached-path hazards
- Parameter, buffer and module ownership at mechanism level
Study, in this order
3 steps
- In-place hazards and the version counterPyTorch Autograd mechanics - in-place operations
In-place operations with autograd: in-place is discouraged because it rewrites the graph instead of allocating new nodes; In-place correctness checks: every tensor keeps a version counter, and accessing a saved tensor whose version changed raises an error
- Detach and the training loop contractD2L - tensors, autograd, MLP and backpropagation
Section 2.5.3 Detaching Computation:
u = y.detach()withz = u * xgives gradient \(u\), not \(3x^2\), while the graph to y persists; Section 2.5.1: the gradient buffer accumulates, sox.grad.zero_()resets it between objectives - Parameters versus buffers`torch.nn.Module` - parameters, buffers and state
Parameters and buffers documentation: modules collect parameters recursively for the optimizer;
register_bufferholds persistent non-parameter tensors that are saved alongside parameters without getting gradients
Focus on
- Every tensor carries a version counter, bumped by each in-place op. Backward saved the counter value too, so mutating a needed value raises an error instead of computing a wrong gradient. Treat the error as a correct alarm, not an obstacle.
- In-place graph rewrites are only safe for updates outside tracking: optimizer steps mutate parameters under no-grad. Anything inside the tracked graph should allocate new nodes.
- Detach cuts provenance silently: gradients stop at the cut and read as correct-looking smaller values (\(u\) instead of \(3x^2\)). A forgotten
zero_()is the mirror bug: values look plausible but sum two objectives. - Ownership: a module owns its parameters recursively (TensorTonic parameter collection below), the optimizer mutates exactly those, and buffers carry persistent non-differentiated state such as running statistics.
Reproduce closed-book
h = x.sigmoid() saves its output; mutate h in place with h.mul_(2) before backward and read the version-counter error. Second, with y = x * x and u = y.detach(), show that z = x * u gives \(dz/dx = u = x^2\), not \(3x^2\). Then state the parameter-versus-buffer rule for a layer with a running counter.Traps
- Silencing the version-counter error by mutating through
.data. Changes made through.dataskip the version counter, sobackwardruns without an error on the mutated value and returns a wrong gradient. Fix the order of operations orclonebefore the mutation instead. - Detaching a hidden activation on the path to the loss. Everything before the cut gets no gradient (
.gradstaysNone) with no error, while later layers still train. Detaching the loss itself does fail loudly; the silent case is the middle of the network. - Registering persistent state as a
Parameter. It then gets gradients and optimizer updates it should never have; buffers travel in state without differentiation.
Practice
TensorTonic: Collect Module Parameters Recursively (autograd-l07-recursive-parameter-collection)
Neural-Network Modules section: Collect Module Parameters Recursively, after Build a Scalar Neuron Module and Evaluate a Multi-Layer Perceptron
Then state for each collected tensor whether the optimizer may mutate it and whether it needs a gradient.
TensorTonic: Apply One Full SGD Training Step (autograd-l08-sgd-training-step)
Losses, Training, and Zeroing Gradients section: Apply One Full SGD Training Step
Confirm gradients are zeroed before the step and the parameter update itself runs outside gradient tracking.
Self-test
-
You modify a saved activation in place, then call
backward. What happens?PyTorch raises a version-counter error whenbackwardaccesses the modified saved tensor. The counter recorded at save time no longer matches. -
For
z = x * uwithu = y.detach()andy = x * x, what is \(dz/dx\)?\(u\), i.e. \(x^2\) in value. Provenance through \(y\) is cut, so the \(2x^2\) path contributes nothing and the total is not \(3x^2\). -
Two
backwardcalls with nozero_(): what is in.grad, and when is that what you want?The sum of both gradients. Wanted when optimizing a sum of objectives; a bug when the calls were meant to be separate batch updates. -
A layer keeps a running event counter alongside its weights.
Parameterorbuffer?Buffer. It must persist and save with the module but never receive gradients or optimizer updates.
Completion criterion
Stop when
You can predict the version error, the detach signature and the accumulation signature on sight, and your module separates parameters from buffers with a stated mutation rule for each.
Mark passed, then open 2.1 Only mark this when you can do it without notes.Leads to
- 2.3 Modules, parameters, buffers and initialization Meet parameters, buffers and init in the framework.
- 2.2 Autograd state: views, in-place ops, leaves and accumulation See views, leaves and in-place ops in autograd state.
Comes after
- 1.6 Tensor layout: shape, stride, views and copiesSee what views plus mutation do to gradients.
- 2.2 Autograd state: views, in-place ops, leaves and accumulationRebuild aliasing contracts inside your own engine.
Resources for this unit
- D2L - tensors, autograd, MLP and backpropagation Use chapter_preliminaries ndarray and autograd, then MLP, backpropagation, and numerical stability and initialization. Recreate scalar operations, affine maps, activation, reduction, stable cross-entropy, topological backward and unbroadcast in a blank file.
- PyTorch Autograd mechanics - in-place operations Read In-place operations with autograd (why in-place rewrites the graph and is discouraged) and In-place correctness checks (the version counter and the error on accessing a modified saved tensor).
- `torch.nn.Module` - parameters, buffers and state Read the parameters and buffers documentation: modules collect parameters recursively for the optimizer, while
register_bufferholds persistent non-parameter tensors saved alongside them.
Also in this section
- Understanding Deep Learning - Chapters 3 to 7 Read Chapters 3-7, emphasizing Chapter 5 for likelihood-derived losses and Chapter 7 for backpropagation and initialization. Use notebooks 3.1, 4.1, 5.1, 6.1 and 7.1 only as equation and plot companions.
- TensorTonic - Build Micrograd from Scratch Use the 25-problem Micrograd study plan in order for derivatives, gradient checks, value graphs, reverse mode, accumulation, operators, modules, losses and training. Preserve local tests as the evidence instead of a platform score.
- Deep-ML candidate - autograd and loss problems After the five-sample platform check passes, use at most one of 26 Basic Autograd Operations, 997 Manual Cross-Entropy Backprop or 1227 Stable Softmax as a changed-prompt drill.
- CS231n Linear Classification - Softmax classifier and numeric stability Read the Softmax classifier section for the cross-entropy form \(L_i = -f_{y_i} + \log \sum_j e^{f_j}\), then the Practical issues: Numeric stability part for the max-subtraction code
f -= np.max(f). - CS231n Neural Networks 3 - Gradient checks Read the Gradient Checks section only: centered formula with \(O(h^2)\) error, relative error with thresholds, double precision, step size \(h\) near 1e-5, kinks, few datapoints, burn-in, regularization and dropout discipline, and checking a few dimensions per parameter.
- PyTorch Broadcasting semantics Read General semantics for the trailing-dimension rules, prepend-1 alignment and max-size result, with the
(5, 1, 4, 1)plus(3, 1, 1)examples; read In-place semantics for why an in-place tensor may not change shape. - `torch.autograd.gradcheck` API and tolerances Read the
gradchecksignature defaults (eps1e-6,atol1e-5,rtol1e-3) and the note that inputs with overlapping memory, such asexpandoutputs, will likely fail the check. - `torch.nn.functional.cross_entropy` - `reduction` and `ignore_index` Read the
ignore_indexandreductionparameters: ignored targets contribute no gradient, and mean reduction averages over the non-ignored targets only. - PyTorch Tensor Views Read the view and storage sharing example (
data_ptr, mutating the view changes the base), the transpose non-contiguous example, the full view-ops list, and the special notes:reshapemay copy,contiguouscopies only when needed, basic indexing returns views while advanced indexing copies. - `torch.Tensor.stride` - stride and storage offset Read the stride definition: strides are per-dimension element steps, and with
storage_offsetthey locate every element. Combine with the Tensor Views page for transposed and expanded layouts.