Unit 4.5 · about 5 h
197 / 324Recurrent gradients, GRU and LSTM gates
Gates are the fix for the gradient pathology of the previous unit. Each gate is a differentiable decision to keep, overwrite or reveal memory.
- derive
- implement
You can now
- Derive why repeated hidden Jacobians vanish or explode from their eigenvalue powers.
- Write the GRU and LSTM updates from memory and explain what each gate preserves or erases.
- Implement both cells from tensor ops and match framework cells on values and gradients.
Covers
- Vanishing and exploding recurrent gradients
- GRU and LSTM gates
Study, in this order
3 steps
- Jacobian powers and gradient pathologyD2L - convolutional and recurrent architecture route
Backprop Through Time in Detail: the linear-recurrence gradient derivation ending in hidden-Jacobian powers. Exercises question 1: symmetric-matrix powers and what they mean for RNN gradients.
- GRU gates and scratch implementationD2L - convolutional and recurrent architecture route
Reset Gate and Update Gate, Candidate Hidden State, Hidden State, and Implementation from Scratch
- LSTM gates and scratch implementationD2L - convolutional and recurrent architecture route
Input, Forget and Output Gates, Memory Cell Internal State with \(C_t = F_t C_{t-1} + I_t \tilde{C}_t\), Hidden State, and Implementation from Scratch
Depth · Skip the JAX
lax.scanbranches unless you train in JAX. The equations are framework-free.
Focus on
- \(dL/dh_t\) fans out through powers of \(W_{hh}\) transposed. Eigenvalues below 1 decay geometrically and above 1 grow geometrically, and tanh or sigmoid derivatives shrink most steps further.
- GRU: \(R\) gates how much past state enters the candidate, and \(Z\) interpolates old state with candidate. LSTM splits the same job across \(F\) for erase, \(I\) for write and \(O\) for reveal, with a separate cell \(C_t\).
- With \(F=1\) and \(I=0\) the cell holds its value exactly and gradients flow unchanged. That constant-error path is the whole point.
- PyTorch and D2L agree on the update gate: \(z\) weights the old state. They differ on the reset gate: D2L multiplies \(R\) into the state before \(W_{hh}\), while
torch.nn.GRUapplies \(r\) after the recurrent matmul and its bias, \(r \cdot (W_{hn} h + b_{hn})\). Implement the PyTorch form when you matchtorch.nn.GRU.
Reproduce closed-book
torch.nn.GRU and torch.nn.LSTM outputs on random input to float32 tolerance, using the PyTorch reset-gate form for the GRU.Traps
- Porting gate formulas between sources without checking conventions. Some texts write \(h_t = (1 - z) h_{t-1} + z \tilde{h}\), the opposite of D2L and PyTorch, and the reset placement differs between D2L and
torch.nn.GRU. - Assuming LSTM always beats GRU. Cost and data decide, which is why the LSTM page asks for the cost comparison.
- Treating saturated gates as a bug to tune away. Gates at 0 or 1 stop gate learning while protecting memory, so that trade is expected.
Practice
Implement gru_cell_forward with torch.nn.GRUCell's [reset, update, new] packing; one test is a GRUCell oracle
The reset gate multiplies only the hidden-side candidate term, the PyTorch form. Then solve LSTM Forward (seq-rnn-lstm-cell-forward) in the same track, with the [input, forget, cell, output] packing.
D2L GRU Exercises 1 and 4: gate values and single-gate ablation
Exercises: question 1 (best reset and update values to use only the input at \(t'\) for the output at \(t\)) and question 4 (a GRU with only a reset gate or only an update gate)
D2L LSTM Exercise 3: gated-cell cost comparison
Exercises: question 3 (computational cost of GRU, LSTM and plain RNN at fixed hidden size, training and inference)
Self-test
-
A linear recurrence has \(W_{hh}\) with spectral radius 1.5. What gradient scale results over 100 steps?
About \(1.5^{100}\), near 4e17: explosion. Below 1 it vanishes the same way, so only near-1 spectra plus gating survive long spans. -
GRU with \(Z=1\) everywhere under the D2L convention. What behavior results?
\(H_t\) equals \(H_{t-1}\): the state freezes and ignores all inputs. With \(Z=0\) the state is just the candidate, a tanh RNN whose recurrent input is gated by \(R\). -
LSTM: which gate settings copy \(C_{t-1}\) unchanged?
\(F=1\) and \(I=0\). The output gate then decides how much of \(\tanh(C)\) shows in \(H\). -
A GRU with a reset gate but no update gate. What breaks?
Every step replaces the state with a fresh tanh candidate, so there is no copy path. Long-span memory must survive repeated multiplication by \(W_{hh}\) and tanh, which brings back vanishing gradients.
Completion criterion
Stop when
Your cells match framework outputs and gradients, and the ablations show which gate carries long-span memory.
Mark passed, then open 4.6 Only mark this when you can do it without notes.Leads to
- 4.6 Hidden state, encoder-decoder models and teacher forcing Track gated memory through encoder-decoder state lifetime.
- 4.7 Decoding, variable-length batches and leak tests Use gate behavior to predict which long spans survive to decoding.
Comes after
- 4.4 Recurrent cells, unrolling and backpropagation through timeExplain why the chained products in this unit force gated cells.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
Also in this section
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.