Unit 2.1 · about 3 h
297 / 324Embeddings, positions and the vocabulary axis
Token IDs select rows; their numerical order carries no meaning. Position representations supply information that a token lookup alone cannot provide.
- explain
You can now
- Explain embedding lookup with a \([V, D]\) table and trace \([B, T]\) token IDs to \([B, T, D]\) vectors.
- Distinguish learned absolute positions, fixed sinusoidal positions and relative-position schemes.
- Explain why computing an encoding beyond the training length does not prove useful length extrapolation.
Covers
- Token embeddings, vocabulary axis and lookup semantics
- Absolute, sinusoidal and relative position vocabulary
Study, in this order
2 steps
- Token embedding lookupModern LLM Notebook - decoder construction path
Section 2, Embedding Lookup, including the end-to-end training demo
Depth · Skip the model configuration survey and section 3's corpus training and t-SNE demo.
- Absolute and sinusoidal positionsModern LLM Notebook - decoder construction path
Sections 1 to 3, then Appendix: Extrapolation of Sinusoidal Position Encoding
Depth · Read the relative-position vocabulary in the appendix; leave interpolation and long-context methods for later.
Focus on
- The vocabulary axis indexes rows. A \([B, T]\) integer tensor gathers vectors without allocating a \([B, T, V]\) one-hot tensor.
- Repeated IDs select identical vectors before position information or contextual layers are applied.
- Learned and sinusoidal encodings are absolute-position representations. RoPE changes \(Q\) and \(K\) rather than adding a position vector to the input.
Reproduce closed-book
[2, 0, 2], and add distinct position vectors. Explain which two vectors match before the addition and why they need not match afterward.Traps
- Treating adjacent token IDs as similar numbers. IDs are labels, not coordinates.
- Calling sinusoidal encoding a guarantee of long-context accuracy. Its formula extends; the learned model's behavior may not.
Practice
Modern LLM Notebook: Exercise 1, Embedding Lookup
Exercises: Exercise 1, Embedding Lookup; complete the lookup and pass its exact-equality assertion
Modern LLM Notebook: Exercise 1, Adding Position Encoding
Exercises: Exercise 1, Adding Position Encoding; add the position vectors and pass the repeated-token assertion
Self-test
-
With \(V=1000\), \(D=32\) and IDs shaped \([2, 5]\), what are the table and output shapes?
The table is \([1000, 32]\), the output is \([2, 5, 32]\), and the table has 32,000 parameters. -
Does moving a token to another position change its raw lookup vector?
No. The same ID selects the same row. Position addition or later contextual processing can change its representation. -
Is sinusoidal encoding relative just because it can express offsets?
No. It assigns a vector to each absolute position. Trigonometric identities let fixed offsets relate those vectors; RoPE instead writes a relative rotation into \(Q-K\) dot products.
Completion criterion
Stop when
Both named exercises pass, and you explain the lookup shapes and the three position families without notes.
Mark passed, then open 2.2 Only mark this when you can do it without notes.Leads to
- 2.2 Scaled dot-product attention from scratch Project the token vectors into queries, keys and values.
- 2.5 RoPE: positions as rotations Put relative-position structure into attention scores.
Comes after
- 1.3 Tokenizer design choices: vocabulary size, Unigram and special tokensAccount for the vocabulary axis in model shapes.
Resources for this unit
- Modern LLM Notebook - decoder construction path Work through English notebooks 03-embedding, 04-position-encoding, 05-transformer-block, 06-mini-gpt and the selected normalization, RoPE, GQA and SwiGLU comparisons in 08-gpt2-to-modern-models. Re-derive shapes and rerun with changed dimensions.
Also in this section
- LLMs from Scratch companion - attention and GPT construction Use the ch03 attention notebooks and ch04 model code. Rebuild the model without a high-level Transformer module and compare one forward and backward path with an independent small reference.
- Stanford CS336 Assignment 1 - model contract Use the named linear, embedding, rmsnorm, positionwise_feedforward, rope, softmax, scaled_dot_product_attention, multihead_self_attention, transformer_block, transformer_lm and transformer_accounting specifications and public tests. Keep implementation independent and respect the course AI and honor-code boundary.
- PyTorch scaled dot product attention reference Use the functional contract only after the explicit attention oracle passes. Transfer identical weights and masks, compare outputs and gradients, and explain dtype, causal and dropout behavior from the documented semantics.
- Deep-ML candidate - three transformer closing drills After the platform quality sample passes, attempt at most problems 490, 53 and 904 as changed-prompt drills for attention shapes and multi-head composition. Preserve independent tests; a judge result does not replace the section checkpoint.
- nanoGPT - residual projection initialization Read
model.py:MLP,Block,GPT.__init__'s special scaled initialization ofc_proj.weight, andGPT._init_weights. Contrast the depth-dependent residual initialization with CS336's component initialization; do not adopt the training or generation code.