Unit 3.4 · about 4 h
188 / 324Regularization: dropout, label smoothing, augmentation and weight decay
Regularization trades training fit for held-out behavior. Each method changes a different part of the objective or the data, and conflating them leads to double-counted or cancelled effects.
- explain
- derive
You can now
- Derive the SGD weight-decay update from the L2-penalized objective and state exactly where the Adam equivalence breaks.
- Explain dropout scaling, label-smoothing targets and augmentation policy as separate generalization levers.
- Choose which parameters escape decay and defend the choice.
Covers
- Dropout, label smoothing and data augmentation
- Weight decay versus an L2 penalty
Study, in this order
5 steps
- Explicit and implicit regularizationUnderstanding Deep Learning - fitting, gradients and regularization
Chapter 9, Sections 9.1 through 9.2 (pp. 138 to 143): L2 regularization and implicit regularization in gradient descent and SGD
- Dropout, label smoothing and augmentationUnderstanding Deep Learning - fitting, gradients and regularization
Chapter 9, Sections 9.3.3, 9.3.4 and 9.3.8 (pp. 147 to 154): dropout, applying noise with label smoothing, and augmentation
Depth · Read Sections 9.3.3, 9.3.4 and 9.3.8 only. Skip Bayesian inference, transfer learning and self-supervised learning.
- Dropout scaling and behaviorD2L - optimization and regularization slice
Dropout in Practice and Implementation from Scratch: inverted scaling and the train and eval split
- Weight decay under SGDD2L - optimization and regularization slice
Norms and Weight Decay and Implementation from Scratch: penalized objective, SGD shrinkage update, and bias exclusion
- Decoupled weight decayPyTorch torch.optim.AdamW reference
AdamW algorithm box: the decoupled decay line and its defaults
Focus on
- L2 adds \((\lambda/2)\|w\|^2\) to the loss, and SGD turns it into per-step shrinkage \((1 - \eta\lambda)\). The \(1/2\) cancels the square's derivative.
- Adam normalizes the whole gradient, L2 term included, by its adaptive denominator, so the penalty no longer shrinks weights by a fixed fraction. AdamW applies the shrinkage outside the adaptive update.
- Inverted dropout divides kept activations by \((1-p)\) during training, so test time needs no scaling and the expectation stays exact.
- Label smoothing replaces hard targets with a blend of the true class and uniform. Augmentation enlarges the data, but its transforms must not change the label.
Reproduce closed-book
Traps
- Adding L2 to Adam and calling it weight decay. The adaptive denominator rescales the penalty per coordinate; use the optimizer's decoupled decay instead.
- Decaying biases and normalization scales by default. They hold offsets, not feature weights; D2L excludes the bias from the penalty, and the same argument covers normalization scales.
- Reading UDL's implicit-regularization equations as exact. The modified loss is a first-order approximation in the step size; it explains tendencies, not exact trajectories.
Practice
Exercises: layer-probability swaps, the dropout versus no-dropout comparison, activation variance plots, and the dropout plus weight-decay interaction
Exercises: the \(\lambda\) sweep on training and validation accuracy and the validation-optimum question
Implement smooth_labels(one_hot, smoothing, num_classes) for one target and for a batch
The true class ends at \(1 - \text{smoothing} + \text{smoothing}/K\) and every other class at \(\text{smoothing}/K\), the same targets torch.nn.CrossEntropyLoss(label_smoothing=...) uses.
Self-test
-
Loss \(L(w)\) plus \((\lambda/2)\|w\|^2\) under SGD with step \(\eta\). What is the update?
\(w \leftarrow w - \eta(\text{grad} + \lambda w) = (1 - \eta\lambda)w - \eta\text{grad}\). -
Why does the same penalty fail to shrink weights by a fixed fraction under Adam?
Adam divides the total gradient, penalty included, by \(\sqrt{v} + \epsilon\). The penalty's effect is rescaled per coordinate by gradient history instead of applying a fixed \((1 - \eta\lambda)\) factor. -
Dropout \(p\) 0.3, activation 10. What values can the next layer see during training, and what is the mean?
0 with probability 0.3, or 10/0.7 = 14.29 with probability 0.7. Mean 0.7 times 14.29 = 10. -
10 classes, smoothing 0.1. What target values do the true class and each other class get?
True class 0.9 + 0.1/10 = 0.91; each other class 0.01.
Completion criterion
Stop when
You write both the SGD shrinkage update and the AdamW decoupled update without notes, and your \(\lambda\) sweep on the D2L regression task shows training loss rising with \(\lambda\) while validation loss first falls, then rises.
Mark passed, then open 3.5 Only mark this when you can do it without notes.Leads to
- 3.5 Optimizers from SGD to AdamW See the decoupled AdamW update that completes the weight-decay story.
- 3.7 Objective checks, tiny-set overfit and shuffled-label controls Label smoothing changes the loss targets; check them there.
Resources for this unit
- Understanding Deep Learning - fitting, gradients and regularization Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
- D2L - optimization and regularization slice Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
- PyTorch torch.optim.AdamW reference Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.
Also in this section
- PyTorch - optimization and automatic mixed precision examples Check optimizer, scheduler, gradient clipping and AMP API semantics. Preserve the correct order for zeroing, scaling, backward, unscaling, clipping, stepping and updating.
- Deep-ML candidate - optimization and normalization problems After the platform admission check, use at most one of 146 Momentum, 169 AdamW, 197 Global-Norm Clipping, 109 LayerNorm or 115 BatchNorm as a changed-prompt drill.
- TrenTorch - training mechanics questions Solve only the questions the units name: Weight Initialization, Layer Normalization, Label Smoothing, Gradient Accumulation, and Warmup and Cosine Decay. Read the tests first; they run in the browser IDE.
- PyTorch GroupNorm reference Read how input channels are split into
num_groupsgroups and which axes each group normalizes; comparenum_groups1 againstnum_channels. - PyTorch RMSNorm reference Read the RMS formula and which axes
normalized_shapecovers; note the missing mean centering and bias. - PyTorch CrossEntropyLoss reference Read Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples; extend the class-index example withignore_indexand a classweight. - Cyclical Learning Rates for Training Neural Networks (Smith, 2017) Read the abstract and the range test: increase the rate linearly for a few epochs and read usable bounds off the loss curve.
- Deep Learning Tuning Playbook FAQ: debugging optimization failures Read How can optimization failures be debugged and mitigated (identifying unstable workloads, warmup, clipping) and Batch normalization implementation details.
- Hands-On Machine Learning with Scikit-Learn and PyTorch Read the Chapter 11 sections named in the units for the PyTorch side of initialization, normalization, clipping and schedules.