Section 7 of 8 · Deep Learning

Attention & Transformer Foundations

Attention must be understood as shape-correct weighted communication with explicit masking and cost. This is the foundation for later LLM work, not a compressed LLM specialization.

7 units 20 topics 0/7

Units

7 in this section

7.1 Queries, keys, values and scaled dot-product attention
derive · implement
4 h
7.2 Causal masks, padding masks and fully masked rows
implement · measure
3 h
7.3 Self-attention, cross-attention and multi-head shapes
implement
4 h
7.4 Positional information and position offsets
derive · implement
3 h
7.5 Transformer blocks: residual stream, normalization and feed-forward
explain · implement
5 h
7.6 Parameter, FLOP and memory accounting for attention
derive · measure
4 h
7.7 Architecture trade-offs and attention correctness tests
explain · measure
5 h

Before moving on

Checkpoint

Derive attention and its cost for a stated shape, implement a general multi-head block, and prove with interventions that neither future nor padded tokens affect valid causal outputs.

Deliberately out of scope

Leave for later

Tokenizer and decoder-only LLM specialization
RoPE, GQA, KV caches and serving
Custom fused attention kernels

Resources for this section

Shared by every unit above

Understanding Deep Learning - Transformers
Simon J. D. Princetheory

primary

Use Chapter 12.1-12.11 and the matching notebooks for dot-product self-attention, extension to transformers, transformer layers and vision transformers.
D2L - attention and transformer implementation route
D2L authorsimplementation

required

Use queries-keys-values, attention-scoring-functions, multihead-attention, self-attention-and-positional-encoding, transformer and vision-transformer. Implement masked single-head attention, multi-head attention and one pre-normalized block before comparing with PyTorch.
PyTorch - scaled dot product attention
PyTorchreference

reference

Use the official function documentation as a differential oracle for shapes, masks, dropout and backend behavior after the manual version passes intervention tests.
Deep-ML candidate - attention problems
Deep-MLpractice

optional

Use 490 Scaled Dot-Product Attention as the fifth platform admission sample. If the judge catches a seeded masking error, use at most one of 490, 491, 53, 904 or 905 as a changed-prompt drill.
TrenTorch - attention and block questions
TrenTorchpractice

support

Solve the multi-head split, sinusoidal positional encoding, causal mask and pre-norm versus post-norm questions the units name, then carry the same checks into your own block.

Apply the section

Mini-project · General multi-head block with an intervention battery (10 to 14 h)

Implement a pre-norm transformer block with multi-head attention, masks and positions from tensor operations, derive its cost for a stated shape, and prove with interventions that future and padded tokens cannot change valid causal outputs.

Steps

Implement scaled dot-product attention with a stable softmax and boolean causal and padding masks. Pass Deep-ML 490 and the CS231n Assignment 3 MultiHeadAttention and PositionalEncoding checks in Transformer_Captioning.ipynb.
Build multi-head attention with an explicit split, transpose and merge, self- and cross-attention through one module, and valid lengths repeated per head in batch-major order.
Assemble a pre-norm block (norm, attention, add, norm, gated MLP, add) with sinusoidal or learned positions. Check shapes across depths, batch sizes and sequence lengths, and identical eval-mode outputs after saving and reloading the state_dict.
Run the intervention battery on the composed block: changing future tokens leaves earlier outputs unchanged, changing padded content leaves valid outputs unchanged, and permuting the batch permutes the outputs. Show that sequence permutation passes without mask or positions and fails with the causal mask.
Differential-test forward and backward against torch.nn.functional.scaled_dot_product_attention on the math backend, within tolerance, with noncontiguous inputs and variable lengths, excluding fully masked rows. Pin your fully-masked-row convention in a separate test.
Derive parameters, forward FLOPs and score memory for \(n = 512\), \(d = 1024\), \(h = 16\) in fp32, then measure the score tensor bytes and a profiler FLOP count and reconcile them under stated conventions.

Deliverables

The attention and block package with masks, positions and the gated MLP.
The test suite: oracle checks, the intervention battery, the SDPA differential tests and the fully-masked-row convention test.
A cost sheet for the stated shape with measured numbers beside derived ones.

Checks

external-tests · Deep-ML 490: Build Scaled Dot-Product Attention
external-tests · CS231n Assignment 3, `Transformer_Captioning.ipynb` attention and positional-encoding checks
rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Attention correctness	Deep-ML 490 and the CS231n attention checks pass.	Forward and backward match the SDPA math backend in `float64` on noncontiguous inputs.
Masks and causality	Future-token and padding interventions leave valid outputs unchanged.	The battery passes on shapes you did not tune for, and the causal sequence-permutation test fails as it should.
Head handling	Split, transpose and merge match a loop over heads.	A test shows that tiling valid lengths with `repeat` masks the wrong heads while `repeat_interleave` masks the right ones.
Block	Shapes hold across depths, and eval-mode outputs survive a save and reload.	You compare pre-norm and post-norm on a short training run and report gradient norms per layer.
Cost accounting	Parameters, FLOPs and score bytes are derived for the stated shape.	Measured score bytes and profiler FLOPs agree with the derivation, with each convention stated, such as multiply-adds versus FLOPs and MiB versus MB.

Feeds into: Capstone artifact 2 can use a model built on this block; the LLM Foundations stage builds its decoder from the same parts.