Unit 8.1 · about 3 h
221 / 324Eager execution, autograd nodes and saved tensors
Every memory number and profiler row later in this section is a consequence of this graph. If you cannot name the saved tensors, you cannot explain what checkpointing or the profiler is measuring.
- explain
You can now
- Trace one eager operation from the Python call to its backward
Functionnode and name each node on the path. - Explain which forward tensors autograd must keep for backward, how long they live, and what frees them.
- Predict whether an in-place edit raises at the edit or at backward, and state what the version numbers mean.
Covers
- Eager execution, dispatch and autograd nodes
- Saved tensors and their lifetimes
Study, in this order
3 steps
- How autograd encodes the historyPyTorch - checkpointing, numerical accuracy and CUDA semantics
How autograd encodes the history: Function graph rebuilt each iteration,
grad_fnentry points, what-you-run-is-what-you-differentiate - Saved tensorsPyTorch - checkpointing, numerical accuracy and CUDA semantics
Saved tensors: the x-squared example,
save_for_backward,_savedattribute inspection, thepowidentity-Trueandexpidentity-Falseexamples with shared storage - In-place checks and saved-tensor hooksPyTorch - checkpointing, numerical accuracy and CUDA semantics
In-place operations with autograd and In-place correctness checks: version counters and the no-error-means-correct-gradients guarantee
Depth · Skim Hooks for saved tensors after the
pack_hookandunpack_hookcontract. Skip the multithreading and complex-number sections.
Focus on
- The graph is rebuilt from scratch every iteration. A
grad_fnobject from a previous iteration is stale and must never be reused. - Forward keeps only what backward needs. For \(x\) squared it keeps \(x\); the rest of the forward temporaries are free to go.
- Object identity of a saved tensor is an implementation detail. The
powcase returns the same object and theexpcase returns a different object, but both share storage and compare equal. - The first backward frees saved tensors. A second backward without
retain_graphfails because the saved values are gone, not because the graph forgot the formula.
Reproduce closed-book
y = (x @ W + b).relu().sum() with a \(2 \times 3\) \(x\), a \(3 \times 4\) \(W\) and a bias of 4, all requiring grad. Walk y.grad_fn through next_functions and write down SumBackward0, ReluBackward0, AddBackward0, MmBackward0 and AccumulateGrad in order. Then show (x ** 2).grad_fn._saved_self is x gives True while its storage is shared.Traps
- Reading
_saved_selfis as identity proof. Packing may return a new object sharing storage, so test storage and equality, never identity. - Editing a leaf with
requires_graderrors at the edit, but editing a saved non-leaf passes and fails at backward with a version error. The two messages describe different checks. - Assuming no-grad regions keep no state and therefore cost nothing. They keep no graph, but the forward compute and its live outputs still cost time and memory.
Practice
The section mini-project covers this: saved-tensor accounting and autograd-state instrumentation for the compact model.
Self-test
-
For
y = (x @ W + b).relu().sum(), name the backward node types fromydown to the leaves.SumBackward0,ReluBackward0,AddBackward0,MmBackward0, thenAccumulateGradat each leaf. -
x.pow(2).grad_fn._saved_self is xisTrue, butx.exp().grad_fn._saved_result is yisFalse. Do both saved tensors share storage with the named tensor?Yes. Both compare equal and share storage. Whether packing returns the same object depends on whether the saved tensor is an output of its owngrad_fn, which is an implementation detail. -
The first
loss.backward()succeeds and the second raises about saved intermediate values. What was freed, and what flag keeps it?The saved tensors were freed after the first backward. Passretain_graph=Trueto keep them for a second backward. -
Backward fails with expected version 0 but found version 1 on a saved tensor. What do the two numbers mean?
The tensor was saved at version 0 and edited in place once since. Gradients computed from the edited value could be wrong, so autograd refuses instead of guessing.
Completion criterion
Stop when
You complete the reproduce walk without notes, and can predict for any small graph which tensors survive until backward and which edit raises where.
Mark passed, then open 8.2 Only mark this when you can do it without notes.Leads to
- 2.5 Custom autograd functions Custom Functions build directly on
save_for_backwardand version rules. - 3.8 Diagnosing failed runs: divergence, stale state, slices and seeds Graph and saved-state reasoning locates stale-state and detachment faults.
Comes after
- 2.5 Custom autograd functionsSee what saved tensors cost at runtime.
Resources for this unit
- PyTorch - checkpointing, numerical accuracy and CUDA semantics Use the official user-guide entries for torch.utils.checkpoint, numerical accuracy, CUDA semantics, reproducibility and serialization. Predict memory categories before measuring and define resume equivalence before running.
Also in this section
- PyTorch - reliability and performance recipes Use the Profiler, Automatic Mixed Precision, model loading and Benchmark recipes, then the Performance Tuning Guide. Apply each to one compact model from an earlier section and keep a CPU fallback when CUDA is unavailable.
- PyTorch - compiler troubleshooting Read graph breaks, guards, recompilations and logging. Compile one stable fragment, intentionally trigger a graph break or recompilation, inspect it, and compare eager and compiled outputs and gradients.
- Deep Learning Training Debugger Use the failure-injection, measurement and evidence rules as the deeper standard. The full Tier 2 project is a later route and is not added to this stage milestone.
- TensorTonic - mixed-precision memory and transformer FLOPs Solve only Mixed-Precision Training Memory Accountant and Transformer Training FLOP Estimator. Reconcile both with the compact model's profiler and document every included and excluded term.
- Learn PyTorch - profiler walkthrough Use notebook 10 only if the official profiler route remains unclear. Transfer the relevant annotations into the compact model instead of keeping the tutorial code.
- TrenTorch - precision, checkpointing and resume Solve Loss Scaling, FP16/BF16 Representable Range, Gradient Checkpointing and Resume-from-checkpoint in NumPy, then reproduce each behavior with the PyTorch API in the compact model.
- Python os - `os.replace` Read
os.replacefor the atomic-rename step of checkpoint writes: save to a temp file in the same directory,flushandfsync, then replace the final name. - Hands-On Machine Learning with Scikit-Learn and PyTorch Read Appendix B, Reduced Precision Models and Mixed-Precision Training, for why
float16training underflows and whatfp32primary weights and loss scaling fix.