Unit 5.5 · about 3 h
204 / 324Patch embeddings and positional information
Patches turn an image into a token sequence. The grid position that convolution got for free must be added back before attention can use it.
- implement
You can now
- Implement patch extraction as stride-equals-size convolution and verify count, order and flat dimension.
- Explain where positional information enters a ViT and why attention needs it.
- Predict token counts and attention-cost scaling when patch size or resolution changes.
Covers
- Image layout, patches and patch embeddings
- Positional information for patch encoders
Study, in this order
3 steps
- Patch embedding as strided convolutionD2L - Vision Transformer
Model plus Patch Embedding sections: \(m = hw/p^2\) patches flattened to \(cp^2\); single convolution with kernel size and stride set to patch size; worked example
img_size96 with patch 16 gives 36 patches of shape(4, 36, 512) - Class token, positions and pre-norm blocksD2L - Vision Transformer
Putting It All Together and Vision Transformer Encoder:
cls_tokenplus learnablepos_embeddingovernum_patches + 1positions;ViTBlockpre-normalization; head readsX[:, 0] - Vision transformers and scaleUnderstanding Deep Learning - representation foundations
Sections 12.10.2-12.10.3 (pp. 229-232): \(16 \times 16\) patches with learned linear map and learned 1D positional encodings; Figure 12.17; SWin multi-scale windows in Figure 12.18
Focus on
- Patchify is a disjoint row-major grid: \((H/p)\) by \((W/p)\) patches, each flattened to \(Cp^2\) values (768 for 3-channel 16px patches).
- One convolution with kernel equal to stride equal to patch size does extraction plus projection; D2L output keeps one vector per patch.
- Self-attention is permutation-equivariant, so order must be injected. D2L learns
pos_embeddingfrom random init; the UDL ViT learns 1D encodings too. - Halving patch size quadruples tokens, multiplies attention-score work by 16, and multiplies per-token projection and MLP work by 4.
Reproduce closed-book
(4, 36, 512) output shape from img_size 96 and patch 16.Traps
- Using overlapping patches by default. ViT and D2L use stride equal to patch size, so patches are disjoint.
- Equating the cls representation with patch averaging. The cls token is learned aggregation; D2L Exercise 2 asks you to implement averaging as an experiment, not an identity.
- Assuming sinusoidal positions. Both D2L and the UDL ViT learn positional embeddings, so 2D layout must be learned from data.
Practice
TensorTonic: Patchify a Batch of Images
Named problem on the free language-modeling-from-scratch plan: split image batches into a row-major grid of non-overlapping flattened patches.
Match D2L patch count and ordering before adding any learned projection.
D2L Vision Transformer exercises
Exercises: img_size versus training time, averaged-patch head instead of cls, and hyperparameter tuning for accuracy.
Self-test
-
\(224 \times 224\) image, patch 16: patches and tokens?
196 patches, 197 tokens with cls. -
Why does a ViT need positional embeddings when a CNN needs none?
Attention is permutation-equivariant (UDL Problem 12.3), so it cannot see order; convolution bakes locality and translation structure into the operation. -
Patch size 16 to 8 at fixed resolution: how does attention work scale?
Tokens quadruple, so the attention matrix grows 16-fold. -
What does
LazyConv2dwith kernel 16 and stride 16 compute on the worked example?Patch extraction plus learned projection to 512-d, giving(4, 36, 512)for batch 4.
Completion criterion
Stop when
Your patchify matches the D2L shape assertions with row-major order, and you predict tokens and attention scaling for a new patch size without notes.
Mark passed, then open 5.6 Only mark this when you can do it without notes.Leads to
- 5.6 Paired modalities: projector heads, alignment and zero-shot retrieval Feed patch-encoded images into the paired image-text objective.
- 7.4 Positional information and position offsets Compare learned patch positions with sequence position schemes.
Comes after
- 4.2 Pooling, transposed convolution, receptive fields and costCompare receptive-field growth with patch tokenization of images.
Resources for this unit
- Understanding Deep Learning - representation foundations Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
- D2L - Vision Transformer Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
Also in this section
- Stanford CS231n - transformer and self-supervised assignment slice Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
- TensorTonic - patchify and symmetric CLIP loss Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
- Deep-ML candidate - InfoNCE Loss If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
- SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
- CLIP - Learning Transferable Visual Models From Natural Language Supervision Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.
- SimSiam - Exploring Simple Siamese Representation Learning Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.