Unit 3.1 · about 5 h
185 / 324Initialization and signal scale across depth
Every later unit assumes signals survive the forward and backward pass. Wrong scale fails silently as flat or exploding loss, and no optimizer setting repairs it.
- derive
- measure
You can now
- Derive the He variance \(2/\text{fan-in}\) for ReLU layers from variance propagation, and state the Xavier form for symmetric activations.
- Predict whether activations and gradients shrink or grow across depth for a stated initialization scale.
- Measure per-layer activation and gradient norms in a deep MLP and name the failing regime.
Covers
- Xavier and Kaiming initialization
- Activation and gradient scale across depth
Study, in this order
3 steps
- Parameter initialization (derivation)Understanding Deep Learning - fitting, gradients and regularization
Chapter 7, Section 7.5 through 7.5.3 (pp. 107 to 111): forward variance, backward variance and the \(4/(\text{fan-in} + \text{fan-out})\) compromise
- Numerical stability, symmetry breaking and XavierD2L - optimization and regularization slice
Vanishing and Exploding Gradients, Breaking the Symmetry, Parameter Initialization and Xavier Initialization; Summary
- HOML: Glorot and He initialization in PyTorchHands-On Machine Learning with Scikit-Learn and PyTorch
Chapter 11, Glorot Initialization and He Initialization (printed pp. 365-368): Equation 11-1, Table 11-1 and the
nn.initcalls withmodel.applyDepth · Read it for the PyTorch default:
nn.Lineardraws Kaiming-uniform weights with bound \(1/\sqrt{\text{fan\_in}}\), a factor \(\sqrt{6}\) below the He bound \(\sqrt{6/\text{fan\_in}}\), so a ReLU stack needsnn.init.kaiming_uniform_orkaiming_normal_to reach the He scale.
Focus on
- ReLU keeps about half of symmetric inputs, so \(\mathbb{E}[h^2]\) is half the pre-activation variance. That half is where the factor 2 in He comes from.
- Xavier balances fan-in against fan-out for the near-linear regime. UDL's compromise averages the two He conditions; it is not the classic tanh Xavier formula.
- Constant initialization never breaks symmetry: identical weights give identical gradients, so units update identically forever. Randomness must differ across units, not just across layers.
- Variance preservation is an expectation over random draws. One unlucky draw still skews a narrow layer, so width matters alongside scale.
Reproduce closed-book
Traps
- Using Xavier variance for ReLU layers. Without the factor 2 the forward variance halves each layer; use He for ReLU and Xavier-style \(1/\text{fan}\) forms for tanh or sigmoid.
- Reading UDL equation 7.34 as classic Xavier. It averages the forward and backward He conditions for ReLU, \(4/(\text{fan-in} + \text{fan-out})\); the chapter Notes (p. 113) say Glorot or Xavier initialization ignores the ReLU and so differs by a factor of two.
- Concluding stability from one seed. Variance statements are expectations, so rerun narrow layers with fresh draws before trusting a scale.
Practice
D2L Numerical Stability and Initialization: Exercises
Exercises: the symmetry-breaking, constant-init and eigenvalue questions
TrenTorch: Weight Initialization
Implement xavier_uniform_bound, xavier_normal_std, kaiming_uniform_bound and kaiming_normal_std from the variance targets
Derive each value before coding: Kaiming normal std with the default gain \(\sqrt{2}\) is \(\sqrt{2/\text{fan\_in}}\), and each uniform bound is \(\sqrt{3}\) times its normal std because \(U(-a, a)\) has variance \(a^2/3\).
Self-test
-
A ReLU layer has fan-in 200. What weight variance preserves activation scale, and why 2 and not 1?
2/200 = 0.01. The ReLU clips about half of symmetric pre-activations to zero, halving \(\mathbb{E}[h^2]\); the 2 compensates that half. -
Tanh layer, fan-in 256, fan-out 128. Give the D2L Xavier uniform bound.
Uniform \(U(-\sqrt{6/384}, \sqrt{6/384}) = U(-0.125, 0.125)\), since 6/(256+128) = 1/64. -
All weights in an MLP layer are initialized to the same constant. What happens?
Every unit computes the same forward value and receives the same gradient, so they update identically forever. Symmetry never breaks; use independent random draws. -
Init variance is far too large. Which two signatures do you expect forward and backward?
Pre-activation magnitudes grow with depth until floating point saturates, and gradient norms grow backward through the same products. Both come from multiplying by large weights at every layer.
Completion criterion
Stop when
You derive \(2/\text{fan-in}\) without notes, and your probe shows collapse below the He value, overflow above it, and bounded drift at it across a few seeds.
Mark passed, then open 3.2 Only mark this when you can do it without notes.Leads to
- 3.2 Dead, saturated and exploding units, and residual paths Match activation and gradient signatures to dead, saturated or exploding regimes.
- 3.3 Batch, layer, group and RMS normalization See how normalization re-centers and re-scales the signals derived here.
Comes after
- 1.6 Eigenvalues, symmetric and positive-semidefinite matricesApply spectral and variance reasoning to signal propagation.
- 2.3 Modules, parameters, buffers and initializationGo deeper on why init scales matter.
Resources for this unit
- Understanding Deep Learning - fitting, gradients and regularization Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
- D2L - optimization and regularization slice Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
- Hands-On Machine Learning with Scikit-Learn and PyTorch Read the Chapter 11 sections named in the units for the PyTorch side of initialization, normalization, clipping and schedules.
Also in this section
- PyTorch - optimization and automatic mixed precision examples Check optimizer, scheduler, gradient clipping and AMP API semantics. Preserve the correct order for zeroing, scaling, backward, unscaling, clipping, stepping and updating.
- Deep-ML candidate - optimization and normalization problems After the platform admission check, use at most one of 146 Momentum, 169 AdamW, 197 Global-Norm Clipping, 109 LayerNorm or 115 BatchNorm as a changed-prompt drill.
- TrenTorch - training mechanics questions Solve only the questions the units name: Weight Initialization, Layer Normalization, Label Smoothing, Gradient Accumulation, and Warmup and Cosine Decay. Read the tests first; they run in the browser IDE.
- PyTorch GroupNorm reference Read how input channels are split into
num_groupsgroups and which axes each group normalizes; comparenum_groups1 againstnum_channels. - PyTorch RMSNorm reference Read the RMS formula and which axes
normalized_shapecovers; note the missing mean centering and bias. - PyTorch torch.optim.AdamW reference Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.
- PyTorch CrossEntropyLoss reference Read Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples; extend the class-index example withignore_indexand a classweight. - Cyclical Learning Rates for Training Neural Networks (Smith, 2017) Read the abstract and the range test: increase the rate linearly for a few epochs and read usable bounds off the loss curve.
- Deep Learning Tuning Playbook FAQ: debugging optimization failures Read How can optimization failures be debugged and mitigated (identifying unstable workloads, warmup, clipping) and Batch normalization implementation details.