Unit 5.6 · about 4 h
205 / 324Paired modalities: projector heads, alignment and zero-shot retrieval
Paired data aligns two geometries into one shared space. That space supports retrieval and zero-shot transfer, but alignment alone provides no reasoning or grounding.
- explain
- implement
You can now
- Explain the symmetric image-text objective over the \(N \times N\) similarity matrix.
- Implement the two-directional loss with learned temperature and batched negatives.
- Run zero-shot retrieval and state the slice limits of the alignment.
Covers
- Projector heads and paired-modality alignment
- Modality imbalance and zero-shot retrieval vocabulary
Study, in this order
4 steps
- Symmetric contrastive objectiveCLIP - Learning Transferable Visual Models From Natural Language Supervision
Section 2.3: \(N \times N\) possible pairings with maximization of the \(N\) real-pair similarities; symmetric cross-entropy; temperature \(\tau\) as a learned log-parameterized scalar
- Scale and temperature limitsCLIP - Learning Transferable Visual Models From Natural Language Supervision
Section 2.5: minibatch 32,768; \(\tau\) initialized to 0.07 and clipped to scale logits by at most 100
- Zero-shot transfer and promptsCLIP - Learning Transferable Visual Models From Natural Language Supervision
Sections 3.1.2 and 3.1.4: zero-shot as text-generated classifier weights;
A photo of a {label}gains 1.3 percent, 80-prompt ensembling adds 3.5 percent - Projector heads and CLIP practiceStanford CS231n - transformer and self-supervised assignment slice
cs231n/simclr/model.pyclassModel: ResNet-50 backbone \(f\) giving the 2048-d feature \(h\), MLP head \(g\) giving \(z\), both returned L2-normalized;CLIP_DINO.ipynb:get_similarity_no_loop,clip_zero_shot_classifier,CLIPImageRetriever.retrieve
Focus on
- The loss makes \(2N\) decisions per batch: each image picks its text and each text picks its image from \(N\) options.
- Temperature is learned, not tuned: log-parameterized, started at 0.07, capped at 100x for stability.
- Each pair trains against 32,767 in-batch negatives, so every step is a 32,768-way proxy classification.
- Zero-shot builds the classifier from text at test time. Prompts are model inputs: the default template gains 1.3 percent and 80-prompt ensembling adds 3.5 percent on ImageNet.
Reproduce closed-book
Traps
- Assuming aligned embeddings understand new words. The text encoder embeds prompts; unseen vocabulary and slices degrade, so alignment quality stays slice-dependent.
- Training one direction only. CLIP optimizes both image-to-text and text-to-image jointly over the same matrix.
- Mixing up the two evaluation spaces. Zero-shot retrieval must use the shared projected space, because that is where images and texts are compared. For a linear probe, also test the features before the projection; SimCLR Section 4.2 found them better in its setup.
Practice
CS231n Assignment 3, Q4: CLIP zero-shot and retrieval
CLIP_DINO.ipynb: get_similarity_no_loop for the \(N \times M\) cosine matrix, clip_zero_shot_classifier for prompt-based labels, CLIPImageRetriever.retrieve with top-\(k\).
Precompute image features once in __init__ instead of per query, as the starter-code comment directs.
Self-test
-
Batch of \(N\) pairs: how many scores and how many decisions?
\(N^2\) scores; \(2N\) decisions, \(N\) per direction. -
Why does CLIP learn \(\tau\), and with what guardrails?
It sets the logit range for the softmax; learning avoids tuning it. Init 0.07, clipped at 100x scaling to prevent instability. -
What is a zero-shot classifier in CLIP terms?
Cosine softmax with L2-normalized inputs, text-generated weights, no bias and temperature scaling; the text encoder generates the classifier. -
Prompt template plus ensembling: what gains on ImageNet?
1.3 percent fromA photo of a {label}, another 3.5 percent from 80-prompt ensembling.
Completion criterion
Stop when
You reproduce 0.2201 by hand, your Q4 functions label and retrieve COCO images, and you can state one slice where you would not trust the alignment.
Mark passed, then open 5.7 Only mark this when you can do it without notes.Leads to
- 5.4 Linear probes, transfer evaluation and pair leakage Judge alignment with frozen probes and leak-checked retrieval splits.
- 7.5 Transformer blocks: residual stream, normalization and feed-forward Open the image and text encoder blocks that feed the shared space.
Comes after
- 5.2 Temperature, negatives and positive constructionReplace augmentation pairs with image-text pairs in the same symmetric loss.
- 5.3 Augmentation shortcuts, false negatives and collapseSee how paired text supervision changes the shortcut landscape.
- 5.4 Linear probes, transfer evaluation and pair leakageEvaluate paired representations with retrieval and zero-shot probes.
- 5.5 Patch embeddings and positional informationFeed patch-encoded images into the paired image-text objective.
Resources for this unit
- Stanford CS231n - transformer and self-supervised assignment slice Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
- CLIP - Learning Transferable Visual Models From Natural Language Supervision Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.
Also in this section
- Understanding Deep Learning - representation foundations Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
- D2L - Vision Transformer Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
- TensorTonic - patchify and symmetric CLIP loss Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
- Deep-ML candidate - InfoNCE Loss If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
- SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
- SimSiam - Exploring Simple Siamese Representation Learning Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.