Unit 5.3 · about 5 h
82 / 324Adaptive optimizers, schedules, clipping and stopping
Adaptive methods carry history as well as parameters. A derivation of that history makes initialization, clipping order and restart errors visible before they affect a training run.
- derive
- implement
You can now
- Derive Adam's zero-initialization bias correction from a finite geometric sum and state its expectation assumptions.
- Distinguish adaptive preconditioning, coupled \(L_2\) penalties and decoupled weight decay.
- Implement clipping, schedule state and optimizer restart with deterministic trajectory checks.
Covers
- Adaptive optimizers, bias correction and weight decay
- Learning-rate schedules, clipping and stopping
Study, in this order
4 steps
- Adaptive preconditioners and bias correctionD2L - Optimization Algorithms
Sections 12.7.2 to 12.7.3 (Adagrad), 12.8.1 (RMSProp), and 12.10.1 to 12.10.2 (Adam), equations 12.10.1 to 12.10.4 and the PyTorch
adamcellDepth · Follow the adjacent Adagrad and RMSProp pages for their algorithms. Skip dataset training and Yogi; second moments are not centered variances.
- Schedule definitionsD2L - Optimization Algorithms
Sections 12.11.2 and 12.11.3.1 to 12.11.3.4: Schedulers, Factor, Multi Factor, Cosine and Warmup
Depth · Read the schedule formulas and run the PyTorch learning-rate assignment and scheduler definitions only. Skip Fashion-MNIST training. Define whether your clock counts updates or epochs.
- Decoupled decay and optimizer statePyTorch optimizer and gradient-clipping contracts
AdamW algorithm box,
epsandweight_decayparameters,state_dictandload_state_dict - Gradient-norm clipping contractPyTorch optimizer and gradient-clipping contracts
clip_grad_norm_: concatenated-gradient norm, in-place modification,max_norm,norm_typeanderror_if_nonfinite
Focus on
- With \(m_0=0\), \(m_t=(1-\beta_1)\sum_{i=1}^t\beta_1^{t-i}g_i\). Constant \(\mathbb{E}[g_i]=\bar{g}\) gives \(\mathbb{E}[m_t]=(1-\beta_1^t)\bar{g}\); constant \(\mathbb{E}[g_i^2]\) gives the analogous second-moment correction. Independence is not needed for these expectations.
- Correction normalizes the finite weights. It does not make \(\hat{m}/\sqrt{\hat{v}}\) an unbiased gradient estimate when gradients change or the denominator is random.
- Clipping projects \(g\) onto a Euclidean ball: \(g\min(1,c/\|g\|_2)\), with zero mapped to zero. Clip before updating moments. AdamW shrinks parameters separately; adding \(\lambda\theta\) to \(g\) changes both moments instead.
- A schedule needs a clock and restart state. A small step caused by a tiny learning rate is not evidence of stationarity; check the un-clipped full-objective gradient and use the residual unit's stopping contract.
Reproduce closed-book
Derive the finite-sum correction by induction, checking \(t=1\). For constant \(g=(2,-1)\), \(\beta_1=0.9\) and \(\beta_2=0.999\), hand-compute \(m_3=(0.542,-0.271)\) and \(v_3=(0.011988004,0.002997001)\); corrected moments are
(2,-1) and (4,1) in exact arithmetic. Check in NumPy float64 with infinity-norm tolerance 1e-12. Then run ten float64 updates on \(f(\theta)=(\theta_1^2+9\theta_2^2)/2\) from (1,-2), \(m_0=v_0=0\), \(\epsilon=10^{-6}\), decay=0.1, \(\mathit{lr}_t=0.01/(1+0.1(t-1))\), \(t=1,\ldots,10\). Clip \(g\) to Euclidean norm 1 using the exact formula above; update moments, then \(\theta=(1-\mathit{lr}_t\cdot 0.1)\theta-\mathit{lr}_t\hat{m}/(\sqrt{\hat{v}}+\epsilon)\). Result: (0.92127176,-1.91402438), allowing 1e-8 in infinity norm for the displayed rounding. Compare torch.optim.AdamW on CPU with float64, the same clipped gradients, betas=(0.9,0.999), eps=1e-6, weight_decay=0.1, foreach=False and the same per-update lr. On PyTorch 2.14.1 the measured infinity-norm difference is 2.22e-16; allow 1e-12. Do not substitute clip_grad_norm_ in this oracle comparison, because its stabilizing denominator changes the mathematical projection slightly. Save \(\theta\),\(m\),\(v\),\(t\) and the schedule clock after step five and check resumed versus uninterrupted final states within 1e-12. No random draws are needed.Traps
- Starting the bias-correction clock at zero, or incrementing it once per coordinate instead of once per update.
- Putting \(\epsilon\) inside the square root while claiming the Adam formula with \(\epsilon\) outside it.
- Restoring parameters but not moments and the schedule clock. The next update is then a different algorithmic state.
Practice
The section mini-project covers this: implement adaptive updates, clipping and update-indexed schedules; test finite-sum moments, checkpoint/resume and an un-clipped stationarity residual on the same deterministic quadratic.
Self-test
-
What assumptions does the bias-correction derivation actually use?
Zero initial moments remove a \(\beta^tm_0\) term. Constant \(\beta\) in \([0,1)\) supplies the finite geometric sum and a nonzero denominator for \(t\ge 1\). Equal first or second moments of the gradient sequence let expectation factor out of the weighted sum. Independence is not required. -
Why is Adam's
vhatnot the gradient variance?It estimates an uncentered second moment. Variance subtracts the square of the mean;vhatdoes not. -
When does coupled \(L_2\) match decoupled decay?
For plain SGD with no momentum or preconditioner, \(\theta-\eta(g+\lambda\theta)\) equals \((1-\eta\lambda)\theta-\eta g\) in exact arithmetic. Adaptive scaling or momentum generally destroys this equivalence. -
What is the norm-clipping solution for \(g=(3,4)\) and \(c=2\)?
Its norm is 5, so the projection is(1.2,1.6). Clamping each coordinate to[-2,2]gives(2,2), which is a different projection and exceeds norm 2.
Completion criterion
Stop when
You can derive the correction with its assumptions and pass moment, decay-placement and restart tests, including the 1e-12 infinity-norm oracle comparison.
Mark passed, then open 5.4 Only mark this when you can do it without notes.Leads to
- 3.5 Optimizers from SGD to AdamW Use the bias and decay derivations beneath the later optimizer comparison.
- 3.6 Schedules, gradient accumulation and clipping Apply schedule clocks and norm clipping to training.
Resources for this unit
- D2L - Optimization Algorithms Use the convexity, GD, SGD, minibatch, momentum, AdaGrad, RMSProp, Adam and scheduler sections for ML derivations and code.
- PyTorch optimizer and gradient-clipping contracts Use AdamW's algorithm and
clip_grad_norm_'s parameters to locate weight decay and clipping precisely. Inspectstate_dictfor optimizer restart; no network training is assigned.
Also in this section
- Stanford EE364a - Convex Optimization Follow lectures 1-9 and 15-16; use lecture 17 only for KKT systems. Leave full interior-point treatment for later.
- Convex Optimization Read Chapters 1-5 and 9 with the course; use Chapter 10 only for equality-constrained Newton steps.
- Fundamentals of Numerical Computation Study §§1.1-1.4, 2.7-2.8, 3.2-3.4 and 4.2-4.3 for floating point, conditioning, stability, least squares and iteration error, in the Python edition.
- CVXPY basic and DCP tutorials Use as the solver oracle after deriving a constrained problem and its KKT conditions by hand.
- Proximal Algorithms Read Sections 1.1, 4.2 and the elementwise soft-thresholding formula in 6.5.2, then the lasso formulation on p. 196. Skip 7.1.1's penalty-scaling and Gram-rewrite errors on p. 197; derive the update from 4.2 and 6.5.2.
- SciPy minimize: BFGS Read the BFGS method description and
gtol,maxiter,c1andc2options. Supply an analytic gradient and compare residuals and evaluation counts, not just iterations. - PyTorch numerical type information Read
torch.finfo'seps,maxandsmallest_normalfields and its subnormal note; inspectfloat64,float32,float16andbfloat16on CPU. - The Accuracy of Floating Point Summation Read Section 2's recursive-sum error bound and Section 3's pairwise and compensated summation algorithms. Leave the statistical error models and historical machine details aside.
- Accurately computing running variance Read the Welford mean and centered-sum recurrences. Prove the centered-sum invariant and translate the recurrence to NumPy; skip the C++ class.