Unit 4.7 · about 5 h
199 / 324Decoding, variable-length batches and leak tests
Decoding and masking turn a trained seq2seq net into honest predictions. One unmasked pad token or one peeking future input invalidates every number above it.
- implement
- measure
You can now
- Implement greedy decoding and describe how beam search trades compute for sequence score.
- Implement masked variable-length loss so padding contributes nothing to loss or metrics.
- Prove with leak tests that padding, hidden-state carry and future tokens cannot change valid outputs.
Covers
- Autoregressive greedy decoding and beam-search recognition
- Variable-length batches, padding and loss masks
- Packed sequences at recognition depth
- Spatial, hidden-state and future-token leak tests
Study, in this order
3 steps
- Masked loss, greedy prediction and BLEUD2L - convolutional and recurrent architecture route
Sections 10.7.5, 10.7.7 and 10.7.8: Loss Function with Masking ((l times mask).sum() over
mask.sum()), greedypredict_step, and the BLEU measureDepth · Use D2L
bleu()as a relative probe only. It weights n-grams by \((1/2)^n\), unlike the uniform \(1/k\) of Papineni et al. - Greedy, exhaustive and beam searchD2L - convolutional and recurrent architecture route
Sections 10.8.1 to 10.8.4: Greedy Search with the 0.048 and 0.054 examples, Exhaustive Search cost, Beam Search, and the length penalty with \(\alpha\) 0.75
- Packed sequences at recognition depthPyTorch sequence packing utilities
pack_padded_sequence: thelengthsargument,enforce_sortedand batch-firstDepth · Recognition only. Know what packing skips and what contract it demands; do not reimplement the packed format.
Focus on
- Greedy decoding feeds each
argmaxback with<bos>start and<eos>stop. Beam search keeps \(k\) hypotheses with length normalization, so larger \(k\) costs linearly more for usually better sequences. - Masked loss divides by valid tokens, not by batch times steps. Dividing by the padded size silently down-weights short-sequence batches.
- Packing skips pad steps in the RNN so pads cost nothing and contribute no state. It needs
lengthsand, by default, length-sorted batches. - A leak test perturbs what should not matter, such as pad values, future tokens or carried state, and asserts that valid outputs do not change: bit-identical for a causal RNN on CPU, and within a tight tolerance wherever a kernel may reorder sums.
Reproduce closed-book
Traps
- Averaging masked loss over all positions including pads. Short sequences then count less per token, so always divide by the mask sum.
- Sorting by length for packing and forgetting to unsort for metrics. Either restore the original order yourself or pass
enforce_sorted=Falseand let PyTorch track the permutation. - Stopping generation at fixed
num_stepsinstead of<eos>. Tail pads then enter BLEU and length statistics as fake content.
Practice
CS231n Assignment 2 Q5: image captioning with vanilla RNNs
Q5: Image Captioning with Vanilla RNNs. Implement vanilla RNNs and apply them to image captioning on COCO in RNN_Captioning_pytorch.ipynb.
D2L beam-search Exercise 2: beam size versus quality and speed
Exercises: question 2 (beam size effects on translation quality and prediction speed in the seq2seq MT problem)
Self-test
-
Batch of lengths 5 and 2 padded to 5. What is the masked-loss denominator?
7: \(5 + 2\) valid tokens. Summing over 10 positions would dilute the short sequence. -
In D2L's greedy-search example, greedy scores 0.048 while picking a different second token gives 0.054. Why does greedy miss it, and what does beam search change?
Greedy takes the most likely token at each step and never revisits. The other second token is less likely by itself but makes the later tokens more likely. Beam search keeps \(k\) partial sequences, so it can keep that branch alive long enough to see the payoff, though it is not guaranteed to find the best sequence. -
The D2L beam score divides the log-probability by \(L^{\alpha}\), with \(\alpha\) 0.75. What does raising \(\alpha\) do?
It favors longer outputs. Log-probabilities are negative and grow in magnitude with length, so a larger divisor for long sequences offsets more of that penalty. \(\alpha\) 0 recovers the raw log-probability, which favors short outputs. -
Leak probe: all
<pad>ids replaced with random ids, and valid outputs change. Is that a bug?Yes. Valid outputs must not depend on pad contents, so the change shows padding leaking into the representation or the loss.
Completion criterion
Stop when
Your masked loss matches a manual per-sequence mean, beam search beats greedy on a held-out sentence, and all three leak probes pass.
Mark passed, then open 5.1 Only mark this when you can do it without notes.Leads to
- 7.2 Causal masks, padding masks and fully masked rows Padding and causal masks generalize these loss masks.
- 7.3 Self-attention, cross-attention and multi-head shapes Cross-attention replaces the fixed context.
Comes after
- 4.5 Recurrent gradients, GRU and LSTM gatesUse gate behavior to predict which long spans survive to decoding.
- 4.6 Hidden state, encoder-decoder models and teacher forcingTurn teacher-forced training into greedy and beam decoding with masks.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.
Also in this section
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.