Gradient checks with finite differences

1. The gradient-check protocol

   CS231n Neural Networks 3 - Gradient checks

   Gradient Checks: the centered formula with \(O(h^2)\) error, relative error with the 1e-2 / 1e-4 / 1e-7 bands, double precision, \(h\) near 1e-5, kinks at ReLU, few datapoints, burn-in point, regularization and dropout discipline, and checking a few dimensions per parameter

   Done
2. Where the formula comes from

   D2L - tensors, autograd, MLP and backpropagation

   The limit definition of the derivative and the linear approximation \(f(x + \epsilon) \approx f(x) + \epsilon f'(x)\): finite differences evaluate the quotient at small nonzero \(\epsilon\) instead of taking the limit

   Depth · Read for the definition only. The centered formula, error orders and protocol all come from the CS231n note above.

   Done
3. Framework defaults and the overlap warning

   `torch.autograd.gradcheck` API and tolerances

   Signature defaults eps 1e-6, atol 1e-5, rtol 1e-3; the note that checked tensors with overlapping memory, for example expand outputs, will likely fail

   Done

Focus on

• Use the centered form \((f(x+h) - f(x-h)) / 2h\). On \(\exp\) at 1 with \(h = 10^{-5}\) it errs by about 6e-11 (second order), while one-sided errs by about 1.4e-5 (first order).
• Compare with relative error \(|a - n| / \max(|a|, |n|)\) and pass both-zero explicitly. CS231n bands: above 1e-2 probably wrong, 1e-4 to 1e-2 uncomfortable, below 1e-4 fine with kinks, 1e-7 or less happy.
• Run in float64: centered differences on \(\exp\) at 1 err by about 1e-2 in float32 against about 6e-11 in float64. Smaller \(h\) is not always better; roundoff grows as truncation shrinks.
• ReLU at exactly 0 gives centered 0.5 against analytic 0, a relative error of 1: the signature of a crossed kink, not a broken derivative. Check few datapoints, past burn-in, with regularization off and noise seeded or disabled.

Reproduce closed-book

Traps

• Using absolute error on tiny gradients. A 1e-4 gap on 1e-5 gradients is a huge failure; relative error exists for exactly this case.
• Checking with regularization on or dropout active. The simple regularization gradient then dominates or noise swamps the comparison, masking a wrong data-loss gradient.
• Perturbing one element of expanded or otherwise overlapping memory. Every aliased index moves at once, so the numerical gradient is meaningless and the check fails on correct code.

Practice

Self-test

1. One-sided differences on \(\exp\) at 1 with \(h = 10^{-5}\) err by about 1.4e-5. What does centered achieve, and why?

   Show answer Got it Missed

2. Analytic gradient is 0 and numerical is 0.5 at a ReLU evaluated at 0. What is the relative error, and what does it mean?

   Show answer Got it Missed

3. Both analytic and numerical gradients are exactly 0. Pass or fail?

   Show answer Got it Missed

4. A correct gradient shows relative error near 1e-2 in float32. What do you change?

   Show answer Got it Missed

5. Why does torch gradcheck warn about expand outputs as inputs?

   Show answer Got it Missed

Leads to

• 2.4 Backward, `zero_grad`, `gradcheck` and evaluation modes Use the framework gradcheck with the same discipline.
• 3.7 Objective checks, tiny-set overfit and shuffled-label controls Extend checks to tiny-set overfit and shuffled-label controls.

Comes after

• 2.6 Backpropagation through dense layers and gradient checksThat unit standardizes centered differences in float64 with \(h = 10^{-5}\) and relative error; this unit's directional test is its matrix-valued companion.
• 1.2 Stable softmax, cross-entropy and loss reductionsCheck the fused loss gradient against finite differences.
• 1.3 Computation graphs and reverse accumulationVerify the engine against finite differences.
• 1.4 Branches, broadcasting and saved values in backwardCheck the branched engine against finite differences.
• 2.4 Backward, `zero_grad`, `gradcheck` and evaluation modesRebuild finite-difference checks in your own engine.

Resources for this unit

• D2L - tensors, autograd, MLP and backpropagation
  D2L authorsrequiredimplementation
  Use chapter_preliminaries ndarray and autograd, then MLP, backpropagation, and numerical stability and initialization. Recreate scalar operations, affine maps, activation, reduction, stable cross-entropy, topological backward and unbroadcast in a blank file.
• CS231n Neural Networks 3 - Gradient checks
  Stanford CS231nreferencereference
  Read the Gradient Checks section only: centered formula with \(O(h^2)\) error, relative error with thresholds, double precision, step size \(h\) near 1e-5, kinks, few datapoints, burn-in, regularization and dropout discipline, and checking a few dimensions per parameter.
• `torch.autograd.gradcheck` API and tolerances
  PyTorchreferencereference
  Read the gradcheck signature defaults (eps 1e-6, atol 1e-5, rtol 1e-3) and the note that inputs with overlapping memory, such as expand outputs, will likely fail the check.