Causal masks, padding masks and fully masked rows

Masks decide which tokens may speak to which. A leak through a future position or a padding slot silently changes the objective the model optimizes.

implement
measure

You can now

Combine causal and padding masks into one score bias and predict each row's valid positions.
Implement valid-length masking and state exactly what a fully masked row returns.
Prove causal behavior with a future-token intervention test rather than by inspecting weights.

Covers

Causal masks, padding masks and all-masked rows

Study, in this order

4 steps

Masked self-attention and interaction matrices
Understanding Deep Learning - Transformers

Section 12.7.2 (pp. 222 to 223): setting dot products to negative infinity before softmax, with Figure 12.12 loss wiring; Section 12.9 with Figure 12.15 (pp. 227 to 228) for encoder versus decoder interaction counts

Done
Masked softmax operation
D2L - attention and transformer implementation route

Section 11.3.2 Masked Softmax Operation: the -1e6 fill, and both runnable examples on (2, 2, 4) inputs with valid lengths [2, 3] and [[1, 3], [2, 4]]

Done
Decoder causal lengths
D2L - attention and transformer implementation route

TransformerDecoderBlock: the training branch building dec_valid_lens rows [1, 2, ..., num_steps], and the inference branch passing None with cached key_values

Done
Mask semantics, causality and dropout
PyTorch - scaled dot product attention

The boolean-mask Note (True means participate, inverse of MultiheadAttention key_padding_mask), the is_causal parameter entry including the non-square upper-left bias, and the dropout Warning that dropout applies in every mode

Done

Focus on

Padding and causal masks compose by intersection: a score survives only if both allow it. D2L valid_lens expresses padding lengths, the decoder \([1..n]\) rows express causality, and a real batch needs both at once.
Mask value conventions differ by API. SDPA bool True means participate while MultiheadAttention key_padding_mask True means mask out. Copying one convention into the other leaks padding.
A fully masked row has no agreed meaning. D2L -1e6 gives uniform weights, all -inf through plain softmax gives NaN, and SDPA on CPU returned zeros in version 2.14. Test your convention instead of assuming one.
SDPA applies dropout_p in every mode. Pass 0.0 outside training or evaluation outputs stay stochastic.

Reproduce closed-book

From a blank file, build combined causal plus padding masks for two sequences of valid lengths 3 and 2 over 4 keys, show every disallowed weight is exactly 0, then change a future token and a padded value and show the earlier valid outputs do not move.

Traps

Trusting uniform weights on a fully masked row. D2L returns \(1/n\) there, so padding values still mix into the output unless you zero the row downstream.
Passing both attn_mask and is_causal to SDPA. The function raises an error; pick one expression of the same mask.
Leaving dropout_p nonzero during evaluation or testing. Outputs differ run to run and differential tests against a manual path fail for no structural reason.

Practice

TrenTorch: Causal mask

Implement build_causal_mask(seq_len) as an additive mask: 0 on and below the diagonal, -inf above it

Then extend it in your own repository: combine it with padding masks for valid lengths 3 and 2 over 4 keys, and decide what a fully masked row returns before softmax turns it into NaN.

Self-test

D2L masked_softmax meets a row with valid length 0 over 4 keys. What weights come out and why is that dangerous?
An SDPA bool mask marks padding as True. What happens, and how does MultiheadAttention key_padding_mask differ?
Why is a future-token intervention a stronger causal proof than reading attention weights?
SDPA is called with dropout_p 0.5 during evaluation. What goes wrong?

Completion criterion

Stop when

Your combined mask gives exact zeros in every disallowed slot, the intervention test passes, and you can state each API mask convention and fully-masked-row behavior from memory.

Mark passed, then open 7.3 Only mark this when you can do it without notes.

Leads to

7.3 Self-attention, cross-attention and multi-head shapes Carry the same mask through folded head batches.
7.5 Transformer blocks: residual stream, normalization and feed-forward Place the mask inside a full residual block.

Comes after

4.7 Decoding, variable-length batches and leak testsPadding and causal masks generalize these loss masks.
7.1 Queries, keys, values and scaled dot-product attentionFeed masked scores into the same row-wise softmax.
7.4 Positional information and position offsetsRecheck valid indices after any position shift.

Resources for this unit

Understanding Deep Learning - Transformers
Simon J. D. Princeprimarytheory
Use Chapter 12.1-12.11 and the matching notebooks for dot-product self-attention, extension to transformers, transformer layers and vision transformers.
D2L - attention and transformer implementation route
D2L authorsrequiredimplementation
Use queries-keys-values, attention-scoring-functions, multihead-attention, self-attention-and-positional-encoding, transformer and vision-transformer. Implement masked single-head attention, multi-head attention and one pre-normalized block before comparing with PyTorch.
PyTorch - scaled dot product attention
PyTorchreferencereference
Use the official function documentation as a differential oracle for shapes, masks, dropout and backend behavior after the manual version passes intervention tests.

Also in this section

Deep-ML candidate - attention problems
Deep-MLoptionalpractice
Use 490 Scaled Dot-Product Attention as the fifth platform admission sample. If the judge catches a seeded masking error, use at most one of 490, 491, 53, 904 or 905 as a changed-prompt drill.
TrenTorch - attention and block questions
TrenTorchsupportpractice
Solve the multi-head split, sinusoidal positional encoding, causal mask and pre-norm versus post-norm questions the units name, then carry the same checks into your own block.