Unit 2.2 · about 4 h
178 / 324Autograd state: views, in-place ops, leaves and accumulation
In-place edits and hidden views are the most common source of corrupt gradients that still run. This unit makes the failure loud and the fix routine.
- explain
- measure
You can now
- Classify an op as view, copy or ambiguous, including basic versus advanced indexing, and predict aliasing with
data_ptr. - Explain leaf versus non-leaf gradient storage, accumulation into
.grad, and whenzero_grad,detachorretain_gradis required. - Trigger, read and repair an in-place version-counter failure without guessing.
Covers
- Views, copies, clone, detach and safe in-place operations
- requires_grad, leaves, nonleaves and gradient accumulation
Study, in this order
4 steps
- Tensor Views: view ops, indexing and reshape ambiguityPyTorch - Autograd mechanics, serialization and reproducibility
View definition and
data_ptrexample; transpose contiguity example; full view-ops list with detach; basic versus advanced indexing note; reshape and contiguous special behavior - Autograd mechanics: saved tensors, `requires_grad`, grad modes, in-place checksPyTorch - Autograd mechanics, serialization and reproducibility
Saved tensors; Setting
requires_grad; Grad Modes table with no-grad versus inference reuse; In-place operations with autograd and In-place correctness checksDepth · Skip Multithreaded Autograd, Autograd for Complex Numbers and the division-by-zero section.
- Autograd: `no_grad`, `detach` and the accumulation demoPyTorch - Learn the Basics
Disabling Gradient Tracking (
no_gradblock anddetach); Optional Reading: Tensor Gradients and Jacobian Products (accumulation across calls, zeroing) - Autograd loop mechanics: `zero_grad` and `no_grad` updatesDeep Learning with PyTorch, 2nd edition - framework foundations
Book Sections 5.5.1 Computing the gradient automatically and 5.5.5 Autograd nits and switching it off (printed pp. 128-132 and 142-144). Notebook cells with
params.grad.zero_(),loss.backward()and thetorch.no_grad()parameter update
Focus on
detach()is a view that shares storage but cuts grad history.clone()is absent from the view-ops list, so it copies the data. Test both claims withdata_ptrandrequires_grad.- Only leaves with
requires_grad=Trueaccumulate into.grad, and everybackward()adds. Non-leaves needretain_gradbefore the backward that consumes them. - Inference-mode tensors cannot re-enter grad-mode computation; no-grad tensors can. Use inference mode only for pure evaluation paths.
- An in-place bump of a saved tensor raises a version-counter error at backward. Clone before the saving op or move the mutation earlier; do not catch and ignore.
Reproduce closed-book
x = torch.randn(4, requires_grad=True) and y = (x ** 2).sum(). Show that x.add_(1) fails at once because x is a leaf that requires grad. Then do the same update under torch.no_grad() and read the version-counter error that y.backward() raises, because pow saved x. Repair it by computing y from z = x.clone(), and check that x.grad equals 2 times the value x had during the forward.Traps
- Treating
detach()as a copy. It shares storage, so an in-place edit through either side is visible to both. - Reading
.gradon a non-leaf. It staysNoneunlessretain_grad()was called; only leaves accumulate. - Updating a leaf that requires grad in place outside
torch.no_grad(). PyTorch refuses at once ("a leaf Variable that requires grad is being used in an in-place operation"); manual parameter updates belong underno_grad.
Practice
The section mini-project covers this: reduce one view or in-place autograd failure to a minimal script, repair it, and keep it as a regression test.
Self-test
-
b = t.view(2, 8);b[0][0] = 3.14. What ist[0][0]?3.14. The view shares storage, so the write is visible through the base. -
Is
t.reshape(2, 8)guaranteed to behave the same way?No. reshape may return a view or a copy, so never rely on aliasing through it. -
You call
out.backward(torch.ones_like(out), retain_graph=True)twice without zeroing. How doesinp.gradafter the second call relate to the first?It is twice the first.backward()adds into.grad; the autogradqs Jacobian-product section shows this doubling, and zeroinginp.gradresets it. -
What is the practical difference between no-grad and inference mode for tensors you keep?
Both skip graph recording, but inference-mode tensors cannot be used in later grad-mode computation while no-grad tensors can.
Completion criterion
Stop when
You classify ten ops as view, copy or ambiguous without the list, and your version-counter script fails before the fix and passes after.
Mark passed, then open 2.3 Only mark this when you can do it without notes.Leads to
- 2.3 Modules, parameters, buffers and initialization Parameters are the leaves you must stop detaching by accident.
- 2.4 Backward, `zero_grad`, `gradcheck` and evaluation modes Turn accumulation and zeroing into a correct loop.
- 1.7 Aliasing, mutation and module ownership Rebuild aliasing contracts inside your own engine.
Comes after
- 1.4 Branches, broadcasting and saved values in backwardSee leaves, views and in-place ops in the framework.
- 1.7 Aliasing, mutation and module ownershipSee views, leaves and in-place ops in autograd state.
- 2.1 PyTorch tensors: creation, indexing, storage and stridesViews share the storage described here; test aliasing against
data_ptr.
Resources for this unit
- PyTorch - Learn the Basics Complete Tensors, Datasets and DataLoaders, Transforms, Build Model, Autograd, Optimization, and Save and Load Model. Rebuild each essential example in a small src package instead of retaining a tutorial notebook.
- PyTorch - Autograd mechanics, serialization and reproducibility Read Autograd mechanics, Serialization semantics and Reproducibility. Test leaf behavior, saved tensors, grad modes, state dictionaries and the exact tolerance promised by one interrupted and resumed run.
- Deep Learning with PyTorch, 2nd edition - framework foundations Read Sections 3.8-3.9 and 5.5 when tensor storage, strides or autograd need a second explanation, with the Chapter 3 and 5 companion notebooks for runnable cells. Inspect the Chapter 4 and 7 code only for a diagnosed data-representation or loading gap.
Also in this section
- Learn PyTorch - selected framework exercises Complete exercise 00, tasks 1-5 in 01, tasks 1-7 in 02, tasks 5-13 in 03, tasks 1-3 in 05, and exercises 1-2 in 07. Recreate solutions independently and verify behavior against current official docs.
- NeetCode ML - PyTorch closing drills After the module gate, solve Pytorch Basics, Layer Normalization, Batch Normalization and RMS Normalization cold. Explain shapes, state and train-evaluation behavior aloud. These do not replace the package and resume test.
- PyTorch Automatic Mixed Precision recipe - Saving/Resuming Read the Saving/Resuming section: save
scaler.state_dict()with the model and optimizer state, at the start of an iteration or afterscaler.update(), and how to resume with or without AMP. - Hands-On Machine Learning with Scikit-Learn and PyTorch Read Chapter 10, Implementing Mini-Batch Gradient Descent Using DataLoaders and Model Evaluation, for the train and evaluate loops and how batch metrics must be aggregated.