Datasets, data loaders and samplers

The loader decides order, batching, dtype and device of every tensor the loop ever sees. Bugs here look like model bugs downstream.

implement

You can now

Implement a Dataset with init, len and getitem plus a collate and transform pipeline, and read batch shapes off the loader.
Control sampler, shuffle, worker count and seeding, and move batches to device with explicit transfer.
State what a loader does not guarantee: ragged batching, worker-shared state and exact resume need extra code.

Covers

Dataset, DataLoader, collate functions and transforms
Sampler, shuffle, worker and device-transfer behavior

Study, in this order

3 steps

Datasets and DataLoaders: FashionMNIST, custom Dataset, batching
PyTorch - Learn the Basics

Loading a Dataset (root, train, download, transform); Iterating and Visualizing; Creating a Custom Dataset (__init__, __len__, __getitem__ with decode_image); Preparing your data with DataLoaders (batch_size, shuffle); Iterate through the DataLoader (batch shapes [64, 1, 28, 28], Samplers link)

Done
Transforms: v2 pipeline and Lambda label encoding
PyTorch - Learn the Basics

ToImage() and ToDtype() with scale=True; Lambda Transforms with one_hot labels

Done
Reproducibility: DataLoader worker seeding
PyTorch - Autograd mechanics, serialization and reproducibility

DataLoader section (per-worker reseeding, worker_init_fn with torch.initial_seed, generator seeding)

Depth · Read only the DataLoader section here; the nondeterminism catalog belongs to the checkpoint unit.

Done

Focus on

Dataset answers one sample; DataLoader answers one batch. Custom datasets implement exactly init, len and getitem.
Default collate stacks equal-shape samples. Ragged samples need a custom collate_fn; no loader pads for you.
shuffle reorders per epoch through the sampler. Workers are subprocesses: each gets a derived seed, and main-process state is not shared with them.
Device transfer is the loop's job. Batches arrive on CPU; .to(device) returns the moved copy.

Reproduce closed-book

From a blank file, write a CSVDataset over (path, label) rows with a transform arg, wrap it in a DataLoader with batch 8, shuffle and 2 workers, and print one batch's shapes. Then add a worker_init_fn that prints each worker seed and show the seeds differ.

Traps

Mutating the dataset object in the main process after iteration starts and expecting workers to see it. Workers hold their own copies; non-persistent workers pick up the change only when the next epoch recreates them.
Creating an RNG object such as np.random.default_rng(0) in the Dataset __init__. Every worker gets an identical copy, so all workers draw the same augmentation stream. The global random, torch and NumPy generators are already reseeded per worker; a private generator must be reseeded in worker_init_fn from torch.initial_seed().
Leaving device transfer implicit. The tutorial batch is CPU [64, 1, 28, 28]; the model page moves inputs explicitly, and .to() must be assigned.

Practice

Learn PyTorch 03: MNIST loading, visualization and DataLoaders (tasks 5 to 13)

Exercises, tasks 5 to 13: load torchvision MNIST train and test, visualize samples, DataLoader with batch_size=32, TinyVGG adaptation, CPU versus GPU timing, predictions, confusion matrix, Conv2d kernel-size probe, FashionMNIST error plots

Skip tasks 1 to 4 (industry prose and overfitting definitions). Tasks 5 to 7 are the loader core; tasks 8 to 13 rehearse the loop of the next unit and can wait for it.

Self-test

What three methods must a custom Dataset implement, and what does each promise?
A DataLoader with two workers is created after torch.manual_seed(0). Do the workers draw identical augmentations, and does the next run repeat this one?
A batch of variable-length sequences hits the default collate. What happens?
The loader yields CPU batches and the model sits on CUDA. What is missing?

Completion criterion

Stop when

Your CSVDataset loader prints correct batch shapes with 2 workers, worker seeds differ on purpose, and a ragged input fails loudly until your collate_fn handles it.

Mark passed, then open 2.7 Only mark this when you can do it without notes.

Leads to

2.7 Explicit training and evaluation loops with correct metrics Feed the loop from a loader you trust.
2.8 Checkpoints, configuration and reproducibility Carry loader position and RNG state into checkpoints.

Resources for this unit

PyTorch - Learn the Basics
PyTorchprimarycourse
Complete Tensors, Datasets and DataLoaders, Transforms, Build Model, Autograd, Optimization, and Save and Load Model. Rebuild each essential example in a small src package instead of retaining a tutorial notebook.
PyTorch - Autograd mechanics, serialization and reproducibility
PyTorchrequiredreference
Read Autograd mechanics, Serialization semantics and Reproducibility. Test leaf behavior, saved tensors, grad modes, state dictionaries and the exact tolerance promised by one interrupted and resumed run.

Also in this section

Learn PyTorch - selected framework exercises
Daniel Bourkerequiredpractice
Complete exercise 00, tasks 1-5 in 01, tasks 1-7 in 02, tasks 5-13 in 03, tasks 1-3 in 05, and exercises 1-2 in 07. Recreate solutions independently and verify behavior against current official docs.
Deep Learning with PyTorch, 2nd edition - framework foundations
Howard Huang, Luca Antiga, Eli Stevens and Thomas Viehmann (Manning)supportimplementation
Read Sections 3.8-3.9 and 5.5 when tensor storage, strides or autograd need a second explanation, with the Chapter 3 and 5 companion notebooks for runnable cells. Inspect the Chapter 4 and 7 code only for a diagnosed data-representation or loading gap.
NeetCode ML - PyTorch closing drills
NeetCodeoptionalpractice
After the module gate, solve Pytorch Basics, Layer Normalization, Batch Normalization and RMS Normalization cold. Explain shapes, state and train-evaluation behavior aloud. These do not replace the package and resume test.
PyTorch Automatic Mixed Precision recipe - Saving/Resuming
PyTorchreferencereference
Read the Saving/Resuming section: save scaler.state_dict() with the model and optimizer state, at the start of an iteration or after scaler.update(), and how to resume with or without AMP.
Hands-On Machine Learning with Scikit-Learn and PyTorch
Aurelien Geron (O'Reilly, 2025)supporttheory
Read Chapter 10, Implementing Mini-Batch Gradient Descent Using DataLoaders and Model Evaluation, for the train and evaluate loops and how batch metrics must be aggregated.