Backward, `zero_grad`, `gradcheck` and evaluation modes

Gradients accumulate silently, grad modes look alike, and eval state hides inside layers. This unit separates the three mechanisms before they combine in the training loop.

implement
measure

You can now

Run backward, autograd.grad and zero_grad correctly, including the grad argument for non-scalar outputs and retain_graph for repeated backward.
Check a handmade op with torch.autograd.gradcheck in float64, and read a tolerance failure as a wrong formula, not noise.
Switch train and eval modes plus no_grad and inference mode correctly, and state what each one changes.

Covers

backward, autograd.grad, zero_grad and gradcheck
no_grad, inference mode and evaluation state

Study, in this order

4 steps

Autograd: computing gradients, `retain_graph`, accumulation
PyTorch - Learn the Basics

Computing Gradients (loss.backward, leaf-only grad, retain_graph); Optional Reading: Tensor Gradients and Jacobian Products (ones seed, accumulation doubling, zeroing)

Done
Zeroing out gradients recipe
PyTorch - Learn the Basics

Introduction (accumulation into .grad); Steps 4 and 5 (optimizer, zero_grad before forward, backward, step); model.zero_grad() equivalence condition

Done
Gradcheck mechanics: central differences and Jacobian rebuild
PyTorch - Autograd mechanics, serialization and reproducibility

Default backward mode gradcheck behavior: Real-to-real functions (central-difference formula, column-by-column numerical rebuild, row-by-row analytical rebuild)

Depth · Skip complex inputs, fast mode and gradgradcheck; the real-to-real section is the whole assignment.

Done
Autograd mechanics: `requires_grad`, grad modes, evaluation mode
PyTorch - Autograd mechanics, serialization and reproducibility

Setting requires_grad (leaf freezing); Grad Modes table with Default, No-grad and Inference rows; Evaluation Mode (nn.Module.eval, orthogonal to grad modes, Dropout and BatchNorm)

Done

Focus on

backward() on a scalar means a ones seed. Non-scalar outputs need an explicit grad argument of matching shape.
The graph is freed after backward unless retain_graph is set. Repeated backward without it errors; with it, .grad keeps accumulating.
gradcheck rebuilds full Jacobians: central differences numerically, one-hot seeds analytically. Run it in float64, as the Extending example does with dtype=torch.double, eps=1e-6, atol=1e-4.
model.eval() changes layer behavior, not grad tracking. Pair it with no_grad for evaluation; inference mode additionally forbids later grad-mode reuse.

Reproduce closed-book

Closed-book, predict the three printed gradients in the autogradqs Jacobian-product snippet (first call, second call, after zeroing) from the eye(4, 5) input, then explain why the second call doubles and the third resets.

Traps

Calling backward twice on one graph. The second call fails unless retain_graph was set, and then it adds to .grad instead of replacing.
Reading model.eval() as gradient control. It is orthogonal to no_grad and inference mode; eval changes Dropout and BatchNorm behavior only.
Assuming model.zero_grad() always equals optimizer.zero_grad(). They match only when every model parameter sits in that optimizer.

Practice

Learn PyTorch 02 task 6: pure-PyTorch `tanh`, then `gradcheck` it

Exercises, task 6 (replicate tanh in pure PyTorch); check the result with torch.autograd.gradcheck in float64 per the gradcheck note

Write tanh from primitives first, match it against torch.tanh, then gradcheck your version. A tolerance failure is a bug in your formula.

NeetCode ML: PyTorch closing drills cold

PyTorch category (4 problems): Pytorch Basics, Layer Normalization, Batch Normalization, RMS Normalization

Solve cold after the module gate. Explain shapes, state and train-versus-eval behavior aloud; these do not replace the reload and resume checks.

Self-test

out = (inp + 1).pow(2).t() with inp = eye(4, 5) requiring grad, seeded with ones. What is inp.grad[0][0] and inp.grad[0][1] after one backward?
The same backward runs again with retain_graph and no zeroing. What changes?
Why does the Extending gradcheck example use dtype=torch.double?
Two evaluation passes over the same test set with the same weights give different accuracies. Which call is missing, and which layers cause the difference?

Completion criterion

Stop when

You predict all three Jacobian-product printouts before running, and your handmade tanh passes gradcheck in float64.

Mark passed, then open 2.5 Only mark this when you can do it without notes.

Leads to

2.5 Custom autograd functions Gradcheck the backward you write yourself.
2.7 Explicit training and evaluation loops with correct metrics Put zeroing, backward and eval switching into a real loop.
1.5 Gradient checks with finite differences Rebuild finite-difference checks in your own engine.

Comes after

1.5 Gradient checks with finite differencesUse the framework gradcheck with the same discipline.
2.2 Autograd state: views, in-place ops, leaves and accumulationTurn accumulation and zeroing into a correct loop.
2.3 Modules, parameters, buffers and initializationDrive the parameters you can now take apart.

Resources for this unit

PyTorch - Learn the Basics
PyTorchprimarycourse
Complete Tensors, Datasets and DataLoaders, Transforms, Build Model, Autograd, Optimization, and Save and Load Model. Rebuild each essential example in a small src package instead of retaining a tutorial notebook.
PyTorch - Autograd mechanics, serialization and reproducibility
PyTorchrequiredreference
Read Autograd mechanics, Serialization semantics and Reproducibility. Test leaf behavior, saved tensors, grad modes, state dictionaries and the exact tolerance promised by one interrupted and resumed run.

Also in this section

Learn PyTorch - selected framework exercises
Daniel Bourkerequiredpractice
Complete exercise 00, tasks 1-5 in 01, tasks 1-7 in 02, tasks 5-13 in 03, tasks 1-3 in 05, and exercises 1-2 in 07. Recreate solutions independently and verify behavior against current official docs.
Deep Learning with PyTorch, 2nd edition - framework foundations
Howard Huang, Luca Antiga, Eli Stevens and Thomas Viehmann (Manning)supportimplementation
Read Sections 3.8-3.9 and 5.5 when tensor storage, strides or autograd need a second explanation, with the Chapter 3 and 5 companion notebooks for runnable cells. Inspect the Chapter 4 and 7 code only for a diagnosed data-representation or loading gap.
NeetCode ML - PyTorch closing drills
NeetCodeoptionalpractice
After the module gate, solve Pytorch Basics, Layer Normalization, Batch Normalization and RMS Normalization cold. Explain shapes, state and train-evaluation behavior aloud. These do not replace the package and resume test.
PyTorch Automatic Mixed Precision recipe - Saving/Resuming
PyTorchreferencereference
Read the Saving/Resuming section: save scaler.state_dict() with the model and optimizer state, at the start of an iteration or after scaler.update(), and how to resume with or without AMP.
Hands-On Machine Learning with Scikit-Learn and PyTorch
Aurelien Geron (O'Reilly, 2025)supporttheory
Read Chapter 10, Implementing Mini-Batch Gradient Descent Using DataLoaders and Model Evaluation, for the train and evaluate loops and how batch metrics must be aggregated.