Unit 4.6 · about 4 h
198 / 324Hidden state, encoder-decoder models and teacher forcing
State lifetime and teacher forcing are silent evaluation bugs. A model that trains on truth and tests on its own guesses fails exactly where the demo looked best.
- explain
You can now
- Explain hidden-state lifetime: what reset, detach and carry choices preserve or leak across batches.
- Explain the encoder-decoder contract from variable-length input to fixed context to variable-length output.
- Explain teacher forcing and exposure mismatch, and predict when each training choice hurts.
Covers
- Hidden-state lifetime and stateful versus stateless evaluation
- Encoder-decoder sequence models
- Teacher forcing and exposure mismatch
Study, in this order
3 steps
- The encoder-decoder interfaceD2L - convolutional and recurrent architecture route
Encoder, Decoder, and Putting the Encoder and Decoder Together: the fixed-state interface with
init_state - Teacher forcing and the encoder contextD2L - convolutional and recurrent architecture route
Sections 10.7.1 and 10.7.2: Teacher Forcing with the
'<bos> Ils regardent .'shift example, and the Encoder context variableDepth · Stop before Section 10.7.4. The training loop, masking and BLEU belong to the next unit.
- Fixed-length batches for variable-length textD2L - convolutional and recurrent architecture route
Section 10.5: fixed-length truncation and padding with valid lengths and the
<bos>-shifted decoder input
Focus on
- The encoder compresses variable-length input into a fixed context. The decoder is a language model conditioned on that context plus its own previous outputs.
- Teacher forcing feeds ground-truth tokens in training but predictions at test time. The mismatch compounds because errors become inputs.
- Reset state between unrelated sequences. Detach stops gradients but keeps values, so a carried state still biases the next batch.
- D2L seeds decoder state directly from encoder state, so layers and hidden size must match. Mismatched shapes need a projection, not a reshape hack.
Reproduce closed-book
('go .', 'va !') with <bos>, <eos> and <pad>, and mark which positions teacher forcing feeds.Traps
- Evaluating statefully after training statelessly, or the reverse. Hidden-state lifetime is part of the model contract, so changing it silently shifts results.
- Reading low teacher-forced loss as generation quality. The loss never scores the model's own continuations, so greedy rollout and BLEU do that in the next unit.
- Padding the decoder input and forgetting the label shift. Decoder input drops the final token and labels drop
<bos>, so misaligning by one trains every position on the wrong target.
Practice
D2L seq2seq Exercises 3 and 4: mismatched decoder init and no-teacher-forcing training
Exercises: question 4 (replace teacher forcing with the previous prediction in training) and question 3 (different encoder and decoder layers or widths for decoder init)
Self-test
-
Why does teacher forcing train faster yet generate worse than its loss suggests?
Training always conditions on truth, so early errors never compound. At test time each error becomes the next input and drift accumulates. -
Hidden state carried across shuffled batches with detach. Is there harm?
Yes. Gradients stop but stale values bias the next unrelated batch, so reset state at sequence boundaries unless order is meaningful. -
Encoder with 2 layers, decoder with 1 layer. Can the D2L init work?
No. It seeds decoder state directly from encoder state, so layers and hidden size must match, or add a projection as seq2seq Exercise 3 asks. -
Target
'a b <eos>'with<bos>start. What are the decoder input and labels?Input'<bos> a b'and labels'a b <eos>'. Same length, shifted by one.
Completion criterion
Stop when
Your shift diagram matches the <bos> and <eos> example, and the no-teacher-forcing run shows how loss and translations change.
Leads to
- 4.7 Decoding, variable-length batches and leak tests Turn teacher-forced training into greedy and beam decoding with masks.
- stage 07 · LLM Foundations Encoder-decoder training prefigures later generation stacks.
Comes after
- 4.4 Recurrent cells, unrolling and backpropagation through timeCarry unrolling and truncation into encoder-decoder training.
- 4.5 Recurrent gradients, GRU and LSTM gatesTrack gated memory through encoder-decoder state lifetime.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
Also in this section
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.