Unit 4.4 · about 5 h
196 / 324Recurrent cells, unrolling and backpropagation through time
Unrolling turns a loop into a deep shared-weight network. BPTT is backprop on that unrolled graph, and truncation decides which dependencies the gradient can ever see.
- derive
- implement
You can now
- Unroll a vanilla RNN cell and derive its per-step shapes and shared-parameter count.
- Derive BPTT as chained Jacobian products and state what truncating at \(\tau\) keeps and drops.
- Implement an RNN language model from tensor ops with gradient clipping and train it below uniform perplexity.
Covers
- Vanilla recurrent cells and sequence unrolling
- Backpropagation through time and truncated BPTT
Study, in this order
3 steps
- The recurrent cell and its unrollingD2L - convolutional and recurrent architecture route
Recurrent Neural Networks with Hidden States (the \(H_t\) recurrence and parameter sharing) and RNN-Based Character-Level Language Models
- Scratch RNN with clipping and decodingD2L - convolutional and recurrent architecture route
The RNN Model from scratch, One-Hot Encoding, Gradient Clipping with the \(\min(1, \theta/\|g\|)\) rule, Training, and Decoding with warm-up prefix
- Backpropagation through time and truncationD2L - convolutional and recurrent architecture route
Analysis of Gradients (full computation with the chained sum, regular truncation at \(\tau\), randomized truncation) and Backprop Through Time in Detail (the \(W_{qh}\), \(W_{hx}\) and \(W_{hh}\) gradient sums)
Focus on
- \(H_t = \phi(X_t W_{xh} + H_{t-1} W_{hh} + b_h)\) reuses the same three matrices every step. Sequence length adds depth, not parameters.
- BPTT sums per-step losses whose gradients chain products of hidden Jacobians. Regular truncation at \(\tau\) drops all paths longer than \(\tau\) and biases long-dependency learning.
- Clip by global norm \(g \gets \min(1, \theta/\|g\|) g\) after unrolling. It bounds the step, not the Jacobian product, so it treats explosion, not vanishing.
- Detaching the hidden state between batches stops gradients but keeps the values. Truncated BPTT still learns only within-segment dependencies.
Reproduce closed-book
Traps
- Averaging the loss over time and thinking truncation is then unbiased. Truncation drops long paths whatever the reduction.
- Carrying hidden state across batches and calling it full BPTT. Values carry but gradients do not unless the graph does.
- Clipping to fix non-learning from vanishing. Clipping only caps large steps, so vanishing needs architecture or objective changes.
Practice
Deep-ML 62: Implement a Simple RNN with Backpropagation Through Time (BPTT)
The RNN forward pass and BPTT gradient in NumPy. The page checks the solution against test cases in the browser.
Do it after the D2L scratch model so the changed prompt tests understanding, not recall.
Self-test
-
\(T=100\), hidden 32. How many \(W_{hh}\) matrices does the unrolled graph hold?
One shared \((32, 32)\) matrix. Unrolling copies the use, not the parameters, and gradients accumulate into it from all steps. -
Truncated BPTT with \(\tau=10\) on a dependency of length 50. What happens?
The gradient path is cut. The update cannot assign credit across the full span, so the dependency is effectively unlearnable at that \(\tau\). -
Gradient norm 12 with \(\theta=1\). What is the clipped gradient?
\(g/12\). \(\min(1, 1/12)\) scales the norm to exactly 1 without changing direction. -
Why does D2L one-hot encode instead of using an embedding layer?
Small char vocab makes one-hot cheap and explicit. It is the same linear map as an embedding row lookup, so shapes and gradients agree.
Completion criterion
Stop when
Your scratch RNN trains below uniform perplexity, Deep-ML 62 passes, and you can state which dependencies your \(\tau\) can and cannot learn.
Mark passed, then open 4.5 Only mark this when you can do it without notes.Leads to
- 4.5 Recurrent gradients, GRU and LSTM gates Explain why the chained products in this unit force gated cells.
- 4.6 Hidden state, encoder-decoder models and teacher forcing Carry unrolling and truncation into encoder-decoder training.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
Also in this section
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.