Unit 3.2 · about 4 h
186 / 324Dead, saturated and exploding units, and residual paths
Most failed runs show one of four signatures. Reading the signature decides whether to change the activation, the scale, the depth structure or the evaluation mode.
- explain
- measure
You can now
- Distinguish dead, saturated, vanishing and exploding regimes from activation histograms and gradient norms.
- Explain how residual identity paths change the gradient product and which failure they leave in place.
- Predict train-time versus test-time behavior differences caused by batch statistics.
Covers
- Dead, saturated, vanishing and exploding units
- Residual paths and gradient flow
Study, in this order
3 steps
- Vanishing and exploding gradients (problem setup)Understanding Deep Learning - fitting, gradients and regularization
Chapter 7, Section 7.5 opening (pp. 107 to 108): small versus large init variance and the vanishing and exploding gradient problem
- Vanishing and exploding gradients, activationsD2L - optimization and regularization slice
Vanishing and Exploding Gradients: the matrix-product gradient form, sigmoid tails versus ReLU, and Breaking the Symmetry
- Residual blocks and batch normalizationUnderstanding Deep Learning - fitting, gradients and regularization
Chapter 11, Sections 11.2 through 11.4 (pp. 189 to 194): residual connections and gradient paths, exploding variance in residual nets, and batch normalization training statistics
Depth · Stop at Section 11.4. Skip the architecture catalog in 11.5 and later.
Focus on
- Sigmoid derivatives peak at 0.25 and decay to zero at both tails; tanh peaks at 1 but still decays. Products of many such terms vanish, while ReLU passes 1 on its active side and 0 elsewhere.
- A residual block outputs \(x + f(x)\), so its gradient is \(I\) plus residual terms. The identity term gives every layer a direct path, but He init still doubles per-block variance, which is why these nets pair residuals with normalization.
- Dead ReLU units output zero and pass zero gradient for all negative inputs. A large negative bias update can silence a unit for the rest of training.
- Batch normalization uses batch statistics during training and frozen statistics at test time. A train and eval gap with normalization present points at state handling before architecture.
Reproduce closed-book
Closed-book, write a two-block residual unrolling \(x_2 = x_0 + f_1(x_0) + f_2(x_0 + f_1(x_0))\) and its gradient with respect to \(x_0\), and circle the identity term that survives even if both residual Jacobians vanish.
Traps
- Calling any flat loss a vanishing-gradient problem. Dead units, saturated outputs, too-small learning rates and broken objectives all flatten curves; check gradient norms before renaming the failure.
- Believing residuals remove all scale problems. UDL Section 11.3 shows per-block variance still doubles under He init; residuals fix paths while normalization fixes scale.
- Testing with training-mode normalization or dropout and reporting the gap as overfitting. Set eval mode first, then compare.
Practice
D2L LeNet with Batch Normalization: learning-rate robustness
LeNet with Batch Normalization; Exercises: the learning-rate comparison and failure-point questions
Self-test
-
In a 10-layer sigmoid MLP, gradient norms shrink by about 0.2 per layer. Where does a factor near 0.2 come from, and what does it do over 10 layers?
Each layer multiplies the gradient by its weights and by a sigmoid derivative of at most 0.25. Ten factors near 0.2 give about \(0.2^{10}\), roughly 1e-7, so early layers barely learn. -
Half the ReLUs in a layer output zero on every training example and their weights never change. Dead or saturated, and why stuck?
Dead. Their pre-activations are negative on every example, so the ReLU derivative is zero and no gradient reaches their incoming weights or bias. They recover only if earlier layers happen to shift their inputs. -
Why does \(x + f(x)\) train deeper than \(f(x)\) alone, in one gradient sentence?
Its Jacobian is \(I\) plus \(df/dx\), so the gradient contains an identity term that reaches early layers directly instead of only through products of small or unstable Jacobians. -
Training accuracy is 95 percent and evaluation accuracy 70 percent, with batch normalization and dropout. Name the first two checks.
First, confirm evaluation runs in eval mode (model.eval(): no dropout, running statistics). Second, measure training accuracy the same way, in eval mode on a training subset, because the logged training number came from train mode. Only a gap that survives both is overfitting.
Completion criterion
Stop when
You classify a dead, a saturated and an exploding layer from histograms and gradient norms alone, and you write the residual gradient with its identity term without notes.
Mark passed, then open 3.3 Only mark this when you can do it without notes.Leads to
- 3.3 Batch, layer, group and RMS normalization Control the scale you just learned to measure.
- 3.8 Diagnosing failed runs: divergence, stale state, slices and seeds Turn these signatures into ordered repair drills.
Comes after
- 3.1 Initialization and signal scale across depthMatch activation and gradient signatures to dead, saturated or exploding regimes.
- 4.3 Equivariance, residual CNN blocks and resolution costPlace residual paths beside dead units and normalization-axis choices.
- 5.7 Graph message passing and oversmoothingTrack embedding health across layers the way you track activations across depth.
Resources for this unit
- Understanding Deep Learning - fitting, gradients and regularization Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
- D2L - optimization and regularization slice Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
Also in this section
- PyTorch - optimization and automatic mixed precision examples Check optimizer, scheduler, gradient clipping and AMP API semantics. Preserve the correct order for zeroing, scaling, backward, unscaling, clipping, stepping and updating.
- Deep-ML candidate - optimization and normalization problems After the platform admission check, use at most one of 146 Momentum, 169 AdamW, 197 Global-Norm Clipping, 109 LayerNorm or 115 BatchNorm as a changed-prompt drill.
- TrenTorch - training mechanics questions Solve only the questions the units name: Weight Initialization, Layer Normalization, Label Smoothing, Gradient Accumulation, and Warmup and Cosine Decay. Read the tests first; they run in the browser IDE.
- PyTorch GroupNorm reference Read how input channels are split into
num_groupsgroups and which axes each group normalizes; comparenum_groups1 againstnum_channels. - PyTorch RMSNorm reference Read the RMS formula and which axes
normalized_shapecovers; note the missing mean centering and bias. - PyTorch torch.optim.AdamW reference Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.
- PyTorch CrossEntropyLoss reference Read Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples; extend the class-index example withignore_indexand a classweight. - Cyclical Learning Rates for Training Neural Networks (Smith, 2017) Read the abstract and the range test: increase the rate linearly for a few epochs and read usable bounds off the loss curve.
- Deep Learning Tuning Playbook FAQ: debugging optimization failures Read How can optimization failures be debugged and mitigated (identifying unstable workloads, warmup, clipping) and Batch normalization implementation details.
- Hands-On Machine Learning with Scikit-Learn and PyTorch Read the Chapter 11 sections named in the units for the PyTorch side of initialization, normalization, clipping and schedules.