Affine maps, nonlinearities and logits

1. Shallow and deep affine structure

   Understanding Deep Learning - Chapters 3 to 7

   Chapter 3, Sections 3.1 to 3.5 (Eq 3.1 to 3.12, printed pp. 25 to 35) for the scalar, multivariate and general affine-plus-activation forms; Chapter 4, Sections 4.1 to 4.4 (Eq 4.1 to 4.16, printed pp. 41 to 49) for composed maps and matrix notation

   Done
2. MLP math and activation curves

   D2L - tensors, autograd, MLP and backpropagation

   Hidden Layers: the linear-collapse derivation \(W = W_1 W_2\), \(b = b_1 W_2 + b_2\) with \(X \in \mathbb{R}^{n \times d}\), \(H \in \mathbb{R}^{n \times h}\); Activation Functions: ReLU, sigmoid and tanh definitions, plots and derivative maxima

   Depth · Skip the Universal Approximators discussion and the Exercises until practice. For the convention that this book calls affine maps linear, read the Chapter 3 Notes paragraph "Linear, affine, and nonlinear functions" in UDL (p. 39).

   Done

Focus on

• Batched affine: \(X\) \((n \times d)\), \(W\) \((d \times h)\), \(b\) \((h,)\) broadcasts over rows, so \(Y = XW + 1 b^\top\) has shape \((n, h)\). The bias gradient is the row sum of the upstream gradient.
• Without a nonlinearity, two affine layers collapse to one: substituting \(H = XW_1 + b_1\) into \(O = HW_2 + b_2\) gives \(W = W_1 W_2\) and \(b = b_1 W_2 + b_2\). Depth without activation adds no function class.
• Sigmoid slope peaks at 0.25 at input 0; tanh peaks at 1 at input 0; both decay toward 0 for large \(|x|\). ReLU has slope 0 below 0 and 1 above; D2L takes the left derivative, 0, exactly at 0.
• GELU is \(x\) times the standard Gaussian CDF, smooth rather than piecewise linear, with \(\operatorname{GELU}(0) = 0\) and slope 0.5 at 0. UDL Figure 3.13 (p. 37) names it Gaussian error linear unit.

Reproduce closed-book

Traps

• Returning \(G\) itself, or its mean, as the bias gradient. The bias is broadcast over the \(n\) rows, so \(dL/db = 1^\top G\), the sum over the batch with shape \((h,)\). A mean belongs only to an explicit mean in the loss.
• Reading ReLU slope at 0 as mathematically settled. Frameworks pick a subgradient (D2L uses 0); the kink matters for gradient checks in a later unit.
• Assuming tanh fixes the sigmoid saturation problem. It is zero-centered but still saturates, and its per-layer factor is at most 1.

Practice

Self-test

1. \(X\) is \((4 \times 3)\), \(W\) is \((3 \times 5)\), \(b\) has 5 entries. What is the shape of \(Y = XW + b\), and of \(dL/dW\) in terms of \(G = dL/dY\)?

   Show answer Got it Missed

2. Why do two stacked affine layers without an activation collapse?

   Show answer Got it Missed

3. What are the maximum slopes of the sigmoid and tanh, and where?

   Show answer Got it Missed

4. What are GELU(0) and the GELU slope at 0?

   Show answer Got it Missed

Leads to

• 1.2 Stable softmax, cross-entropy and loss reductions Feed logits into a stable softmax and cross-entropy.
• 1.3 Computation graphs and reverse accumulation Differentiate affine chains with the chain rule.

Resources for this unit

• Understanding Deep Learning - Chapters 3 to 7
  Simon J. D. Princeprimarytheory
  Read Chapters 3-7, emphasizing Chapter 5 for likelihood-derived losses and Chapter 7 for backpropagation and initialization. Use notebooks 3.1, 4.1, 5.1, 6.1 and 7.1 only as equation and plot companions.
• D2L - tensors, autograd, MLP and backpropagation
  D2L authorsrequiredimplementation
  Use chapter_preliminaries ndarray and autograd, then MLP, backpropagation, and numerical stability and initialization. Recreate scalar operations, affine maps, activation, reduction, stable cross-entropy, topological backward and unbroadcast in a blank file.