Unit 5.1 · about 3 h
200 / 324Embedding geometry and contrastive logits
Contrastive objectives compare directions, not positions. If norms leak into the logits, the loss rewards loud embeddings instead of aligned ones.
- derive
You can now
- Explain why cosine similarity keeps angles and discards norms, and when that helps retrieval.
- Derive NT-Xent as softmax cross-entropy over the \(2N-1\) other views, with \(1/\tau\) logit scaling.
- Predict the loss for degenerate batches, including the \(\log(2N-1)\) uniform case.
Covers
- Embeddings, dot products, cosine geometry and normalization
- Metric learning and contrastive logits
Study, in this order
2 steps
- NT-Xent and cosine similaritySimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 2.1, Eq. (1): \(\operatorname{sim}(u,v)\) as the L2-normalized dot product and \(l_{i,j}\) with the sum over \(k \ne i\); Algorithm 1 for the full loop
- Why normalization mattersSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 5.1, Table 5: L2 Yes versus No rows, with the sentence that normalization plus temperature weights examples by hardness
Depth · Skip the temperature-value comparison; the next unit owns \(\tau\).
Focus on
- Cosine similarity is a dot product after L2 normalization. Positive scaling of either vector changes nothing.
- NT-Xent treats the \(2N-1\) other views as classes. The positive stays in the denominator, so the loss is a valid cross-entropy.
- With identical embeddings every row is uniform at any \(\tau\), so the loss equals \(\log(2N-1)\): \(\log 3 = 1.0986\) for \(2N = 4\).
- Without normalization, one large-norm embedding dominates every softmax row it appears in.
Reproduce closed-book
Traps
- Calling cosine similarity translation-invariant. It is invariant to positive scaling, not to shifts; centering the batch changes all angles.
- Dropping the positive from the denominator. SimCLR sums over \(k \ne i\), which includes the positive; without it the loss can go negative.
- Reading NT-Xent as fixed-class classification. The classes are the current batch views, so the same pair gets a different loss in different batches.
Practice
Deep-ML 384: Contrastive Loss (InfoNCE / SimCLR-style)
Implement contrastive_loss(embeddings, temperature). The visible tests check two identical pairs at \(\tau = 1.0\) for 0.5514 and four identical embeddings for 1.0986.
Then add local tests for temperature extremes, a missing positive, duplicate pairs, and normalized versus unnormalized inputs.
Self-test
-
Embeddings \(\left[\begin{smallmatrix}1&0\end{smallmatrix}\right]\), \(\left[\begin{smallmatrix}1&0\end{smallmatrix}\right]\), \(\left[\begin{smallmatrix}0&1\end{smallmatrix}\right]\), \(\left[\begin{smallmatrix}0&1\end{smallmatrix}\right]\) at \(\tau = 1\) give loss 0.5514. Show the per-row terms.
For row 0 the positive scores 1 and the two negatives score 0, so \(l = -1 + \log(e + 2) = 0.5514\). Rows 1, 2 and 3 match by symmetry, and the mean over all four rows is 0.5514. -
All four embeddings are identical. What is the loss at \(\tau = 1\)?
\(\log 3 = 1.0986\). Every row is uniform over its three candidates, so each row contributes \(\log 3\). -
Why does SimCLR Table 5 favor L2 normalization?
Raw dot products mix angle with norm, so loud vectors dominate the softmax. With normalization plus temperature the loss weights examples by relative hardness, and Top-1 rises from 57.2 to 64.4 at the best setting. Without normalization the contrastive accuracy is higher (91.7 versus 87.8) yet the representation is worse, so the pretext score does not measure representation quality. -
Is the positive pair part of its own denominator?
Yes. The sum runs over all \(k \ne i\) including the positive index \(j\), which keeps the loss a proper cross-entropy bounded below by 0.
Completion criterion
Stop when
You derive NT-Xent without notes, reproduce 0.5514 by hand, and your implementation passes the Deep-ML tests plus the four local edge tests.
Mark passed, then open 5.2 Only mark this when you can do it without notes.Leads to
- 5.2 Temperature, negatives and positive construction Scale the logits you derived here with temperature and test batch-size effects.
- 1.2 Stable softmax, cross-entropy and loss reductions Implement the log-sum-exp and reduction the denominator needs.
Resources for this unit
- SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
Also in this section
- Understanding Deep Learning - representation foundations Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
- D2L - Vision Transformer Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
- Stanford CS231n - transformer and self-supervised assignment slice Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
- TensorTonic - patchify and symmetric CLIP loss Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
- Deep-ML candidate - InfoNCE Loss If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
- CLIP - Learning Transferable Visual Models From Natural Language Supervision Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.
- SimSiam - Exploring Simple Siamese Representation Learning Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.