Section 2 of 8 · Deep Learning
PyTorch Foundations & Low-Level Training Loops
PyTorch is the execution substrate for the remaining stage. This module makes tensor storage, autograd state, data flow and checkpoint ownership explicit before higher-level architectures are added.
8 units 19 topics 0/8
Units
8 in this section
- 2.1 PyTorch tensors: creation, indexing, storage and strides explain · implement3 h
- 2.2 Autograd state: views, in-place ops, leaves and accumulation explain · measure4 h
- 2.3 Modules, parameters, buffers and initialization explain · implement4 h
- 2.4 Backward, `zero_grad`, `gradcheck` and evaluation modes implement · measure3 h
- 2.5 Custom autograd functions derive · implement3 h
- 2.6 Datasets, data loaders and samplers implement4 h
- 2.7 Explicit training and evaluation loops with correct metrics implement · measure4 h
- 2.8 Checkpoints, configuration and reproducibility implement · measure5 h
Before moving on
Checkpoint
Present a minimal trainer without a high-level framework, explain every owned state item, resume under a stated tolerance, and reduce and repair one view or in-place autograd failure.
Deliberately out of scope
Leave for later
- Lightning, Accelerate and other trainer abstractions
- Distributed and sharded checkpoints
- Production input pipelines
Resources for this section
Shared by every unit above
- PyTorch - Learn the Basics Complete Tensors, Datasets and DataLoaders, Transforms, Build Model, Autograd, Optimization, and Save and Load Model. Rebuild each essential example in a small src package instead of retaining a tutorial notebook.
- PyTorch - Autograd mechanics, serialization and reproducibility Read Autograd mechanics, Serialization semantics and Reproducibility. Test leaf behavior, saved tensors, grad modes, state dictionaries and the exact tolerance promised by one interrupted and resumed run.
- Learn PyTorch - selected framework exercises Complete exercise 00, tasks 1-5 in 01, tasks 1-7 in 02, tasks 5-13 in 03, tasks 1-3 in 05, and exercises 1-2 in 07. Recreate solutions independently and verify behavior against current official docs.
- Deep Learning with PyTorch, 2nd edition - framework foundations Read Sections 3.8-3.9 and 5.5 when tensor storage, strides or autograd need a second explanation, with the Chapter 3 and 5 companion notebooks for runnable cells. Inspect the Chapter 4 and 7 code only for a diagnosed data-representation or loading gap.
- NeetCode ML - PyTorch closing drills After the module gate, solve Pytorch Basics, Layer Normalization, Batch Normalization and RMS Normalization cold. Explain shapes, state and train-evaluation behavior aloud. These do not replace the package and resume test.
- PyTorch Automatic Mixed Precision recipe - Saving/Resuming Read the Saving/Resuming section: save
scaler.state_dict()with the model and optimizer state, at the start of an iteration or afterscaler.update(), and how to resume with or without AMP. - Hands-On Machine Learning with Scikit-Learn and PyTorch Read Chapter 10, Implementing Mini-Batch Gradient Descent Using DataLoaders and Model Evaluation, for the train and evaluate loops and how batch metrics must be aggregated.
Apply the section
Mini-project · Minimal PyTorch trainer with owned state (8 to 12 h)
Write a small trainer in plain PyTorch, with no high-level training framework, for one compact model and dataset. Every piece of state is explicit, checkpointed and tested: data order, parameters, optimizer, scheduler, random number generators and metrics.
Steps
- Define one config object for data, model, optimizer, scheduler, seed and paths, and log it with a hash at the start of every run.
- Build the
DatasetandDataLoaderwith a seeded generator and aworker_init_fn, and record split provenance (indices or file hashes) so train, validation and test can be rebuilt exactly. - Write explicit train and eval loops: switch
train()andeval(), evaluate undertorch.no_grad(), placezero_graddeliberately, and accumulate metrics by sample count rather than averaging per batch. - Add one custom autograd
Function, such as a scaledtanhor aclampwith a straight-through gradient, and passtorch.autograd.gradcheckinfloat64. - Checkpoint model, optimizer, scheduler, epoch, step, RNG states and data-order state. Resume from disk mid-run and compare with an uninterrupted run under a stated tolerance.
- Reduce one view or in-place autograd failure to a minimal script, such as an in-place edit of a tensor saved for backward, repair it, and keep the script as a regression test.
Deliverables
- The trainer package: config, data, model, loops and checkpointing.
- Tests:
gradcheckfor the customFunction, resume equivalence, metric aggregation with an uneven last batch, and the in-place regression script. - A state inventory: every piece of state the run owns, where it lives, and whether the checkpoint restores it.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Owned state | The inventory lists model, optimizer, scheduler, RNG, data order and metrics, and the checkpoint restores each one. | A test drops each item from the checkpoint in turn and shows which resume check then fails. |
| Resume | Resume from disk matches the uninterrupted run within a stated tolerance. | On CPU it matches exactly, and you name the settings that relies on, such as deterministic algorithms, worker count and generator state. |
| Evaluation correctness | Evaluation runs in eval mode under no_grad, and metrics weight by sample count. | A test with an uneven last batch shows a per-batch average differs while your metric does not. |
| Custom `Function` | gradcheck passes in float64. | You also break the backward on purpose and show gradcheck catches it. |
| Autograd failure | One minimal failing script, its repair and a regression test. | You name the version-counter or view rule involved and show a second variant that fails at a different point, at the edit or at backward. |
Feeds into: Capstone artifact 2, the reproducible training run: this trainer runs the model from section 4, 6 or 7.