Unit 3.8 · about 5 h
192 / 324Diagnosing failed runs: divergence, stale state, slices and seeds
Loss curves alone never name the broken mechanism. Diagnosis is a designed experiment per failure, and this unit turns the section checkpoint into a repeatable drill.
- measure
You can now
- Localize flat, diverging, NaN and eval-gap failures to objective, optimization, state or data causes using competing controls.
- Instrument the first invalid value and per-slice errors, and report seed variation instead of single-run conclusions.
- Repair a failing compact run, add a regression test, and defend the configuration with measured evidence.
Covers
- Nonlearning, divergence, NaN and Inf diagnosis
- Train-evaluation mismatch and stale state
- Learning curves, per-slice errors and seed variation
Study, in this order
4 steps
- Error sources and learning curvesUnderstanding Deep Learning - fitting, gradients and regularization
Chapter 8, Sections 8.1 through 8.3 (pp. 118 to 126): training a simple model, sources of error, and reducing error
Depth · Read Sections 8.1 through 8.3. Skip double descent in 8.4 here.
- Overfitting, interpolation and early stoppingD2L - optimization and regularization slice
Revisiting Overfitting and Regularization, Early Stopping, and Classical Regularization Methods for Deep Networks
- Debugging optimization failuresDeep Learning Tuning Playbook FAQ: debugging optimization failures
FAQ How can optimization failures be debugged and mitigated: Identifying unstable workloads, Potential fixes, Learning rate warmup, Gradient clipping
- Train and eval state revisitedD2L - optimization and regularization slice
Batch Normalization During Prediction and Dropout in Practice, revisited with a stale-state lens: frozen statistics, eval mode, and what a mode gap proves
Focus on
- Flat from the start means no learning signal: check objective wiring and LR scale before touching the architecture. Diverging or NaN means too much update: halve the LR, enable clipping, then bisect to the first invalid op.
- An eval gap that vanishes in eval mode was stale state, not overfitting. Compare modes before comparing curves.
- Aggregate metrics hide slice failures. Cut errors by source, length or class, and rerun the fix on a second seed; one seed proves nothing about stability.
- Patience-based early stopping needs a no-improvement threshold and a fixed budget. Without both, stopping decisions are hindsight.
Reproduce closed-book
Closed-book, write the five-branch drill: flat, diverging, NaN or Inf, eval gap, noisy plateau. For each, name the first measurement, the control that splits two causes, and the fix you try first.
Traps
- Reading any falling training loss as progress. Memorization falls too; only held-out slices and controls tell fitting from learning.
- Fixing NaN by switching to
float64or adding an \(\epsilon\) somewhere. Either can hide the first invalid op; find it first, then decide between learning rate, clipping, initialization and objective order. - Reporting the best of many seeds as the result. Report the spread across at least two seeds, with the same seeds for the baseline and the fix, before claiming a fix.
Practice
The section mini-project covers this: fault-injection diagnosis of a failing compact run with regression tests across two seeds.
Self-test
-
Loss is exactly flat from step 0 with a nonzero learning rate. Name three causes and the two measurements that split them.
Gradients are zero (broken or detached graph, dead units); gradients are nonzero but parameters do not move (optimizer built on other parameters,zero_gradbetween backward and step, a missing step); or parameters move but the loss ignores them (loss computed from a detached tensor or from the targets). Log the gradient norm and the parameter-update norm per step. -
Loss explodes to NaN at step 200 after falling. What is the ordered response?
Halve the LR and rerun to test update scale; enable global-norm clipping; log the first step whose activations or gradients go non-finite and inspect that op's inputs. Change one thing per rerun. -
Eval error exceeds train error only when dropout and batch norm are in training mode. Diagnosis?
Stale evaluation state, not overfitting. Eval must use frozen statistics with dropout off; the gap measures mode mismatch until modes are fixed. -
A fix works on seed 0 but fails on seed 1. What do you conclude and what next?
The fix is not established; seed variation dominates. Rerun both configs on at least two more seeds and compare means and spreads before keeping it.
Completion criterion
Stop when
Given a fresh failing run, you name the branch, run its control, repair it and show the fix holding on a second seed.
Mark passed, then open 4.1 Only mark this when you can do it without notes.Leads to
- 2.7 Explicit training and evaluation loops with correct metrics Fix stale state with correct train and eval loops and metrics.
- 8.5 Overflow, underflow and finding the first invalid value Trace the first invalid value with framework instrumentation.
Comes after
- 2.8 Checkpoints, configuration and reproducibilityUse resume equivalence as a debugging control.
- 3.2 Dead, saturated and exploding units, and residual pathsTurn these signatures into ordered repair drills.
- 3.3 Batch, layer, group and RMS normalizationDebug stale train and eval normalization state.
- 3.5 Optimizers from SGD to AdamWTurn stalled or diverging optimizer behavior into ordered repair drills.
- 3.7 Objective checks, tiny-set overfit and shuffled-label controlsTurn control failures into ordered repair drills.
- 8.1 Eager execution, autograd nodes and saved tensorsGraph and saved-state reasoning locates stale-state and detachment faults.
- 8.5 Overflow, underflow and finding the first invalid valuePromote first-invalid evidence into full divergence diagnosis.
- 8.7 Complete checkpoints, atomic writes and reproducibility boundariesUse resume equivalence as a control when a restarted run diverges.
Resources for this unit
- Understanding Deep Learning - fitting, gradients and regularization Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
- D2L - optimization and regularization slice Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
- Deep Learning Tuning Playbook FAQ: debugging optimization failures Read How can optimization failures be debugged and mitigated (identifying unstable workloads, warmup, clipping) and Batch normalization implementation details.
Also in this section
- PyTorch - optimization and automatic mixed precision examples Check optimizer, scheduler, gradient clipping and AMP API semantics. Preserve the correct order for zeroing, scaling, backward, unscaling, clipping, stepping and updating.
- Deep-ML candidate - optimization and normalization problems After the platform admission check, use at most one of 146 Momentum, 169 AdamW, 197 Global-Norm Clipping, 109 LayerNorm or 115 BatchNorm as a changed-prompt drill.
- TrenTorch - training mechanics questions Solve only the questions the units name: Weight Initialization, Layer Normalization, Label Smoothing, Gradient Accumulation, and Warmup and Cosine Decay. Read the tests first; they run in the browser IDE.
- PyTorch GroupNorm reference Read how input channels are split into
num_groupsgroups and which axes each group normalizes; comparenum_groups1 againstnum_channels. - PyTorch RMSNorm reference Read the RMS formula and which axes
normalized_shapecovers; note the missing mean centering and bias. - PyTorch torch.optim.AdamW reference Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.
- PyTorch CrossEntropyLoss reference Read Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples; extend the class-index example withignore_indexand a classweight. - Cyclical Learning Rates for Training Neural Networks (Smith, 2017) Read the abstract and the range test: increase the rate linearly for a few epochs and read usable bounds off the loss curve.
- Hands-On Machine Learning with Scikit-Learn and PyTorch Read the Chapter 11 sections named in the units for the PyTorch side of initialization, normalization, clipping and schedules.