Unit 7.7 · about 5 h
220 / 324Architecture trade-offs and attention correctness tests
Attention fails quietly: transposed axes still run, padding still mixes, fused kernels still return. Only deliberate tests separate a correct block from a plausible one.
- explain
- measure
You can now
- Compare convolution, recurrence and attention by invariance, path length, parallelism, state and scaling, and pick the simplest adequate mechanism.
- Run an intervention battery (future-token, padding, permutation) that proves mask and position semantics.
- Diagnose axis, contiguity and dtype hazards in a multi-head implementation.
- State which behaviors need measurement (backend, dtype, length regime) versus derivation.
Covers
- Convolution, recurrence and attention trade-offs
- Head, sequence and batch axis hazards
- Causal, padding and low-precision correctness tests
- Noncontiguous and variable-length input behavior
Study, in this order
3 steps
- Architecture comparison and hazard sitesD2L - attention and transformer implementation route
Comparing CNNs, RNNs, and Self-Attention with its figure, plus the
transpose_qkvandtranspose_outputpair and thevalid_lensrepeatin the multi-head implementation for the axis checklist - Transformer summary and long-sequence galleryUnderstanding Deep Learning - Transformers
Section 12.11 (p. 232) for parallel computation with quadratic scaling; Section 12.9 gallery (pp. 227 to 228) for sparsification patterns at recognition depth; Section 12.10 (pp. 228 to 231) for convolutional inductive bias versus data scale
Depth · Stop at recognition of the gallery patterns; sparse and image architectures are not built here.
- Backend selection and determinism limitsPyTorch - scaled dot product attention
Backend selection functions, the outputs-may-differ paragraph, and the cuDNN nondeterminism Note with its deterministic flag
Focus on
- Choose by mechanism, not fashion: CNNs share local filters with translation structure, RNNs carry fixed state through \(O(n)\) sequential steps, attention pays \(O(n^2 d)\) for content-based \(O(1)\) paths. The D2L table makes the trade exact.
- The axis checklist is short: which axis the softmax sums to 1, whether head splitting precedes the transpose, whether
valid_lensrepeats per head, and whether transposed tensors are contiguous before anyview. - Interventions beat inspections: perturb future tokens, change padded content, permute the batch (and the sequence, for an unmasked block without positions), and require matching outputs. UDL Section 12.7.2 explains how a decoder that sees the future can cheat; only the intervention proves yours cannot.
- Some claims need a run: fused-backend numerics, half-precision intermediates, and the length where materialization binds. Derive the model, then measure the regime.
Reproduce closed-book
Traps
- Asserting bitwise equality across backends. The docs say outputs may differ by backend and that cuDNN may pick a nondeterministic algorithm; tolerance plus a pinned backend is the valid test.
- Using
viewon a freshly transposed tensor. Head-fold transposes are noncontiguous;viewfails wherereshape, or contiguous plusview, succeeds. - Reading causality off one layer weights. Dropout, residuals and later layers remix paths; the intervention on the composed block is the proof.
Practice
The section mini-project covers this: the intervention battery, variable-length and noncontiguous stress cases, and the SDPA differential test with the fully-masked-row convention pinned.
Self-test
-
Fill the D2L trade-off row for CNNs, RNNs and self-attention: compute order, sequential ops, max path.
CNN: \(O(k n d^2)\), \(O(1)\), \(O(n/k)\). RNN: \(O(n d^2)\), \(O(n)\), \(O(n)\). Self-attention: \(O(n^2 d)\), \(O(1)\), \(O(1)\). -
Which D2L line protects multi-head masking, and what is the failure without it?
torch.repeat_interleave(valid_lens, num_heads, dim=0). Without it D2Lmasked_softmaxfails with a mask shape error. Tiling withrepeatinstead runs but gives heads another example's length, because the fold is batch-major. -
Why must the SDPA differential test exclude fully masked rows?
Conventions disagree there: D2L-1e6gives uniform weights, all-infthrough plain softmax givesNaN, and SDPA on CPU returned zeros in PyTorch 2.14. Assert your convention in a separate test. -
transpose_qkvends withreshapeafterpermute. Why notview?Permuted tensors are noncontiguous, andviewrequires contiguity.reshapecopies when needed;viewwould raise. Call contiguous first if a laterviewmatters.
Completion criterion
Stop when
The battery passes on shapes you did not tune for, including all-padded rows under your stated convention, and you can attribute any mismatch to mask, axis, dtype or backend.
Mark passed, then open 8.1 Only mark this when you can do it without notes.Leads to
- 8.9 Profiling training phases and verifying optimizations Profile the verified block phase by phase.
- stage 07 · LLM Foundations Carry the verified block into decoder specialization.
Comes after
- 4.3 Equivariance, residual CNN blocks and resolution costCompare convolutional inductive bias against attention trade-offs.
- 7.5 Transformer blocks: residual stream, normalization and feed-forwardStress-test the block you just built.
- 7.6 Parameter, FLOP and memory accounting for attentionTurn the cost model into architecture choices and tests.
Resources for this unit
- Understanding Deep Learning - Transformers Use Chapter 12.1-12.11 and the matching notebooks for dot-product self-attention, extension to transformers, transformer layers and vision transformers.
- D2L - attention and transformer implementation route Use queries-keys-values, attention-scoring-functions, multihead-attention, self-attention-and-positional-encoding, transformer and vision-transformer. Implement masked single-head attention, multi-head attention and one pre-normalized block before comparing with PyTorch.
- PyTorch - scaled dot product attention Use the official function documentation as a differential oracle for shapes, masks, dropout and backend behavior after the manual version passes intervention tests.
Also in this section
- Deep-ML candidate - attention problems Use 490 Scaled Dot-Product Attention as the fifth platform admission sample. If the judge catches a seeded masking error, use at most one of 490, 491, 53, 904 or 905 as a changed-prompt drill.
- TrenTorch - attention and block questions Solve the multi-head split, sinusoidal positional encoding, causal mask and pre-norm versus post-norm questions the units name, then carry the same checks into your own block.