Optimizers from SGD to AdamW

1. Optimizers from GD to Adam

   Understanding Deep Learning - fitting, gradients and regularization

   Chapter 6, Sections 6.1 through 6.4 (pp. 77 to 90): gradient descent, SGD, momentum with Nesterov, and Adam with bias correction

   Done
2. Momentum (heavy-ball form)

   D2L - optimization and regularization slice

   The Momentum Method and Effective Sample Weight: heavy-ball update and the \(\eta/(1-\beta)\) effective step; Theoretical Analysis

   Done
3. AdaGrad recognition and failure regime

   D2L - optimization and regularization slice

   Sparse Features, Preconditioning and The Algorithm: accumulating denominator and its decay

   Done
4. RMSProp recognition and failure regime

   D2L - optimization and regularization slice

   The Algorithm: leaking average fix for the unbounded denominator

   Done
5. Adam with bias correction

   D2L - optimization and regularization slice

   The Algorithm and Yogi: bias correction and the divergence fix

   Done
6. AdamW state and update equations

   PyTorch torch.optim.AdamW reference

   AdamW algorithm box: decoupled decay step, moments, corrections and defaults

   Done

Focus on

• UDL writes momentum as an exponential average of gradients, while D2L and PyTorch use heavy-ball \(v \leftarrow \beta v + g\). Same direction, different scale: heavy-ball steps larger by \(1/(1-\beta)\).
• AdaGrad divides by the root of all past squared gradients, so its steps decay toward zero even far from the optimum on dense problems. RMSProp's leaking average keeps the denominator current.
• Adam debiases both moments because zero initialization drags early averages toward zero. For \(\beta_1 = 0.9\) the correction fades within a few dozen steps; for \(\beta_2 = 0.999\) it still multiplies \(v\) by about 10 at step 100 (\(1 - 0.999^{100}\) is about 0.095) and fades only after a few thousand steps.
• AdamW applies weight decay before the adaptive step, outside \(m\) and \(v\). The decay rate then means the same thing at every coordinate regardless of gradient scale.

Reproduce closed-book

Traps

• Mixing UDL and PyTorch momentum hyperparameters. \(\beta\) 0.9 means an effective step 10 times larger under heavy-ball than under UDL's averaged form at the same \(\eta\).
• Carrying a learning rate between SGD and Adam. Adam moves each coordinate by about lr regardless of gradient scale, while an SGD step scales with the gradient, so a good SGD rate says nothing about a good Adam rate.
• Tuning Adam's weight_decay as if it were L2 strength. In plain Adam the penalty enters the moments; switch to AdamW for decoupled meaning.

Practice

Self-test

1. Heavy-ball momentum \(\beta\) 0.9, \(\eta\) 0.1, constant gradient \(g\). What is the steady-state step?

   Show answer Got it Missed

2. Why does AdaGrad stall on dense deep-learning objectives, and what does RMSProp change?

   Show answer Got it Missed

3. Adam's first step on gradient 2 with default betas and \(\eta\) 0.001 is about 0.001 regardless of gradient size. Show why.

   Show answer Got it Missed

4. AdamW weight_decay 0.01, lr 0.001. How much does a weight of 0.5 shrink per step from decay alone?

   Show answer Got it Missed

Leads to

• 3.6 Schedules, gradient accumulation and clipping Schedule the learning rate and scale the batch around the optimizer you just built.
• 3.8 Diagnosing failed runs: divergence, stale state, slices and seeds Turn stalled or diverging optimizer behavior into ordered repair drills.

Comes after

• 5.2 Gradient descent, stochastic gradients and momentumApply these update and state tests to neural-network training.
• 5.3 Adaptive optimizers, schedules, clipping and stoppingUse the bias and decay derivations beneath the later optimizer comparison.
• 3.4 Regularization: dropout, label smoothing, augmentation and weight decaySee the decoupled AdamW update that completes the weight-decay story.

Resources for this unit

• Understanding Deep Learning - fitting, gradients and regularization
  Simon J. D. Princeprimarytheory
  Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
• D2L - optimization and regularization slice
  D2L authorsrequiredimplementation
  Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
• PyTorch torch.optim.AdamW reference
  PyTorchreferencereference
  Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.