Section 3 of 8 · Deep Learning
Training Dynamics, Regularization & Diagnosis
A working model is not enough. You need controls that distinguish an objective, optimization, data, mode, numerical or capacity failure before changing the configuration.
8 units 21 topics 0/8
Units
8 in this section
- 3.1 Initialization and signal scale across depth derive · measure5 h
- 3.2 Dead, saturated and exploding units, and residual paths explain · measure4 h
- 3.3 Batch, layer, group and RMS normalization derive · implement4 h
- 3.4 Regularization: dropout, label smoothing, augmentation and weight decay explain · derive4 h
- 3.5 Optimizers from SGD to AdamW derive · implement5 h
- 3.6 Schedules, gradient accumulation and clipping implement · measure5 h
- 3.7 Objective checks, tiny-set overfit and shuffled-label controls implement · measure3 h
- 3.8 Diagnosing failed runs: divergence, stale state, slices and seeds measure5 h
Before moving on
Checkpoint
Given an unfamiliar failing compact run, choose a discriminating control, repair the causal defect, add a regression test, and defend the configuration with activation, gradient, update and generalization evidence across two seeds.
Deliberately out of scope
Leave for later
- Optimizer research and broad tuning sweeps
- Distributed optimizer state
- Task-specific augmentation libraries
Resources for this section
Shared by every unit above
- Understanding Deep Learning - fitting, gradients and regularization Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
- D2L - optimization and regularization slice Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
- PyTorch - optimization and automatic mixed precision examples Check optimizer, scheduler, gradient clipping and AMP API semantics. Preserve the correct order for zeroing, scaling, backward, unscaling, clipping, stepping and updating.
- Deep-ML candidate - optimization and normalization problems After the platform admission check, use at most one of 146 Momentum, 169 AdamW, 197 Global-Norm Clipping, 109 LayerNorm or 115 BatchNorm as a changed-prompt drill.
- TrenTorch - training mechanics questions Solve only the questions the units name: Weight Initialization, Layer Normalization, Label Smoothing, Gradient Accumulation, and Warmup and Cosine Decay. Read the tests first; they run in the browser IDE.
- PyTorch GroupNorm reference Read how input channels are split into
num_groupsgroups and which axes each group normalizes; comparenum_groups1 againstnum_channels. - PyTorch RMSNorm reference Read the RMS formula and which axes
normalized_shapecovers; note the missing mean centering and bias. - PyTorch torch.optim.AdamW reference Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.
- PyTorch CrossEntropyLoss reference Read Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples; extend the class-index example withignore_indexand a classweight. - Cyclical Learning Rates for Training Neural Networks (Smith, 2017) Read the abstract and the range test: increase the rate linearly for a few epochs and read usable bounds off the loss curve.
- Deep Learning Tuning Playbook FAQ: debugging optimization failures Read How can optimization failures be debugged and mitigated (identifying unstable workloads, warmup, clipping) and Batch normalization implementation details.
- Hands-On Machine Learning with Scikit-Learn and PyTorch Read the Chapter 11 sections named in the units for the PyTorch side of initialization, normalization, clipping and schedules.
Apply the section
Mini-project · Seeded-fault diagnosis of a compact run (10 to 14 h)
Instrument a small MLP or CNN trained with your section 2 trainer, inject faults you do not choose yourself, and diagnose each with a discriminating control, a repair, a regression test and evidence across two seeds. Finish by defending one configuration with measurements.
Steps
- Instrument the run: per-layer activation mean and std, dead or saturated fractions, gradient norms, update-to-weight ratios, loss, and train and validation metrics, logged per step.
- Run the controls first and keep them as tests: overfit one fixed batch to near-zero loss, and train on shuffled labels until training accuracy rises while held-out accuracy stays at chance.
- Write a fault script that applies one fault from a seeded list and seals the choice until diagnosis is committed. Cover a
softmaxbeforeCrossEntropyLoss, missingzero_grad, eval mode left on, an excessive learning rate, bad init scale and augmented validation data. - Diagnose three injected faults. For each, write rival hypotheses first, run one control that separates them, repair, and add a regression test that fails on the fault and passes on the repair.
- Compare SGD with momentum and AdamW with decoupled weight decay at matched budgets, with warmup, a cosine schedule and gradient clipping, across two seeds.
- Check gradient accumulation: \(k\) micro-batches of size \(b\) match one batch of size \(k\) times \(b\) within tolerance for a model without batch normalization or dropout, and show what breaks when the loss is not divided by \(k\).
Deliverables
- The instrumented trainer, the control tests and the fault script.
- Three postmortems, each with symptom, rival hypotheses, discriminating control, repair, regression test and one limitation.
- A configuration defense: two-seed curves with activation, gradient, update and validation evidence for the chosen optimizer and schedule.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Controls | The fixed batch reaches near-zero loss, and shuffled labels leave held-out accuracy at chance. | Both run as automated CPU tests. |
| Diagnosis | Each of three faults is found with a stated discriminating control. | Each postmortem names at least two rival hypotheses and shows why the control separates them. |
| Regression guards | Each repair has a test that fails on the fault. | The tests run on both seeds and check a mechanism, such as a norm, a mode flag or a reduction, not only the final loss. |
| Configuration evidence | Two seeds for the chosen configuration, with activation, gradient and update statistics. | Update-to-weight ratios and the validation gap support the chosen learning rate and decay against one rejected alternative. |
| Accumulation | Accumulated gradients match the large batch within a stated tolerance. | You also show the mismatch caused by an undivided loss and explain why batch normalization breaks exact equivalence. |
Feeds into: Capstone artifact 3, the diagnosis and profile casebook: the objective or data postmortem and the numerical or state postmortem can come from here.