Unit 5.2 · about 4 h
201 / 324Temperature, negatives and positive construction
Temperature and batch size set the difficulty of the discrimination game. The positive construction declares which variation the representation must ignore.
- derive
- measure
You can now
- Derive the \(1/\tau\) sharpening of logits and gradients, and predict saturation at small \(\tau\).
- Explain how batch size sets the negative count and when longer training closes the gap.
- Build positives by augmentation composition and state the invariance each transform declares.
Covers
- Temperature and batch-negative effects
- Positive and negative construction
Study, in this order
3 steps
- Temperature and normalizationSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 5.1, Table 5: \(\tau\) 0.05, 0.1, 0.5 and 1.0 with L2 normalization; best Top-1 64.4 at \(\tau = 0.1\); the hard-negative weighting sentence
- Batch size and training lengthSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Sections 2.2 and 5.2 with Figure 9: batch sizes 256 to 8192 with 16,382 negatives per positive at 8192; larger batches help early and the gap shrinks with longer training
- Augmentation compositionSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 3.1 with Figures 5 and 6: single transforms fail while crop plus color distortion wins; color histograms alone suffice as a shortcut
Focus on
- Dividing by \(\tau\) multiplies every logit gap. Small \(\tau\) sharpens toward \(\operatorname{argmax}\); large \(\tau\) flattens toward uniform.
- The positive-logit gradient is \(-(1 - p_{\text{pos}})/\tau\). Easy pairs saturate near zero gradient at small \(\tau\), so hard negatives drive learning.
- Negatives per positive equal \(2(N-1)\): 510 at batch 256 and 16,382 at batch 8192. Longer training closes much of the batch gap.
- Random crop plus color distortion wins because it blocks the color-histogram shortcut, not because either transform is strong alone.
Reproduce closed-book
Traps
- Pushing \(\tau\) ever lower. Below 0.1 easy pairs saturate and a few hard negatives dominate; Table 5 drops from 64.4 at 0.1 to 59.7 at 0.05.
- Assuming bigger batches always win. Figure 9 shows large gains at 100 epochs that mostly disappear with longer training on randomly resampled batches.
- Letting labels choose positives and still calling the method self-supervised. SimCLR positives are two views of one image; once class labels pick positives, it is supervised contrastive learning and must be compared as such.
Practice
TensorTonic: Implement Symmetric CLIP Contrastive Loss
Named problem on the free language-modeling-from-scratch plan: implement the two-directional loss over an image-text similarity batch.
Transfer your NT-Xent derivation: CLIP swaps augmentation pairs for image-text pairs and averages both softmax directions.
Self-test
-
Similarities \(\left[\begin{smallmatrix}2&1&0\end{smallmatrix}\right]\) at \(\tau = 1\): what is the positive probability?
\(e^2/(e^2 + e + 1) = 0.665\). The remaining mass splits 0.245 and 0.090. -
What happens to easy-pair gradients as \(\tau\) goes to 0?
The softmax saturates to one-hot, so \((1 - p)/\tau\) goes to 0 for decided pairs and learning concentrates on the remaining hard pairs; too small stalls training. -
Batch 8192 gives how many negatives per positive, and why?
16,382. The batch holds \(2N = 16{,}384\) views; excluding the anchor and its positive leaves \(2N - 2\) negatives. -
Positive-identification accuracy is near perfect but linear evaluation is poor. What went wrong?
The task is solvable by shortcut, such as color histograms under crop-only augmentation. Composition makes pairing harder and representations better.
Completion criterion
Stop when
You predict \(\tau\)-sharpening and batch-size effects from the formulas, and your two-directional implementation passes the page tests.
Mark passed, then open 5.3 Only mark this when you can do it without notes.Leads to
- 5.3 Augmentation shortcuts, false negatives and collapse Test which augmentations create shortcuts and how collapse shows in the embeddings.
- 5.6 Paired modalities: projector heads, alignment and zero-shot retrieval Replace augmentation pairs with image-text pairs in the same symmetric loss.
Comes after
- 5.1 Embedding geometry and contrastive logitsScale the logits you derived here with temperature and test batch-size effects.
Resources for this unit
- SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
Also in this section
- Understanding Deep Learning - representation foundations Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
- D2L - Vision Transformer Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
- Stanford CS231n - transformer and self-supervised assignment slice Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
- TensorTonic - patchify and symmetric CLIP loss Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
- Deep-ML candidate - InfoNCE Loss If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
- CLIP - Learning Transferable Visual Models From Natural Language Supervision Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.
- SimSiam - Exploring Simple Siamese Representation Learning Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.