Positional information and position offsets

1. Positional encoding

   Understanding Deep Learning - Transformers

   Section 12.3.1 (pp. 213 to 214): permutation equivariance, absolute encoding matrix, and relative per-offset parameters, with Figure 12.5

   Done
2. Positional encoding implementation and theory

   D2L - attention and transformer implementation route

   Positional Encoding: the sine-cosine formula, the PositionalEncoding class with max_len 1000 and dropout after the add, the encoding_dim 32 with num_steps 60 run plotting columns 6:10, the binary-frequency intuition, and the relative-offset rotation derivation

   Done
3. Order-free self-attention

   D2L - attention and transformer implementation route

   The paragraphs before Positional Encoding on parallel computation discarding sequential order

   Done

Focus on

• Without positions the block is permutation-equivariant: \(\operatorname{Sa}[XP]\) equals \(\operatorname{Sa}[X]P\). UDL Problem 12.3 asks for this proof; it is the reason order must be injected as input.
• Absolute encodings add a unique column per position. The D2L TransformerEncoder scales embeddings by \(\sqrt{\text{width}}\) before the add because the encodings stay in \([-1, 1]\).
• Sinusoid pairs rotate with offset: position \(i\) plus \(\delta\) is a fixed \(2 \times 2\) rotation of position \(i\), depending on \(\delta\) and frequency only. Shifting text shifts encodings linearly.
• Relative schemes index by the offset between key and query positions, not by absolute index. They survive absolute shifts that absolute tables must relearn.

Reproduce closed-book

Traps

• Not stating where positions enter. UDL lists three placements: once at the input, at every layer, or only in the query and key computation. The last leaves values free of position, which changes what later layers can read; write down which one your block uses.
• Exceeding D2L max_len 1000. The stored table is fixed width, so longer inputs fail on the add; rebuild the table instead of truncating silently.
• Comparing encodings or outputs in train mode. D2L applies dropout to the sum of embeddings and encodings, so train-mode runs drop entries at random; compare in eval mode or with dropout 0, as the D2L plot does.

Practice

Self-test

1. State permutation equivariance for self-attention and explain why it forces positional input.

   Show answer Got it Missed

2. Write the D2L sinusoidal encoding entries.

   Show answer Got it Missed

3. What does the \(2 \times 2\) rotation from position \(i\) to position \(i\) plus \(\delta\) depend on?

   Show answer Got it Missed

4. A 1500-token input meets D2L default PositionalEncoding. What happens?

   Show answer Got it Missed

Leads to

• 7.5 Transformer blocks: residual stream, normalization and feed-forward Add positions before the first residual block.
• 7.2 Causal masks, padding masks and fully masked rows Recheck valid indices after any position shift.

Comes after

• 5.5 Patch embeddings and positional informationCompare learned patch positions with sequence position schemes.

Resources for this unit

• Understanding Deep Learning - Transformers
  Simon J. D. Princeprimarytheory
  Use Chapter 12.1-12.11 and the matching notebooks for dot-product self-attention, extension to transformers, transformer layers and vision transformers.
• D2L - attention and transformer implementation route
  D2L authorsrequiredimplementation
  Use queries-keys-values, attention-scoring-functions, multihead-attention, self-attention-and-positional-encoding, transformer and vision-transformer. Implement masked single-head attention, multi-head attention and one pre-normalized block before comparing with PyTorch.