Unit 5.3 · about 5 h
202 / 324Augmentation shortcuts, false negatives and collapse
Without negatives the objective accepts a constant answer; with lazy positives it accepts a cheating one. Both failure modes look like falling loss.
- explain
- measure
You can now
- Explain how augmentation shortcuts and false negatives arise, using the color-histogram case.
- Detect collapse with per-dimension spread, norm and rank checks.
- Explain why stop-gradient and predictor asymmetry prevent collapse in positive-only methods.
Covers
- Augmentation assumptions, false negatives and shortcuts
- Representation collapse and norm diagnostics
Study, in this order
3 steps
- Self-supervision and augmentationUnderstanding Deep Learning - representation foundations
Sections 9.3.7-9.3.8 (pp. 152-153): contrastive pairs of transformed images; augmentation as label-preserving expansion; Figure 9.12
- Negative rule and augmentation shortcutsSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 2.1: all other \(2(N-1)\) views count as negatives, including same-class views; Section 3.1 with Figure 6: color histograms alone suffice to distinguish images
- Collapse without stop-gradientSimSiam - Exploring Simple Siamese Representation Learning
Figure 2: 67.7 percent with stop-gradient versus 0.1 percent without, plus the per-channel spread plot; Table 1 row (a): no predictor also gives 0.1 percent
Focus on
- An augmentation declares an invariance. Crop-only declares color histograms fair game, and the encoder takes that shortcut.
- Every batch negative is repelled, including same-class images. At batch 8192 with 1000 balanced classes, each anchor repels about 16 same-class views.
- Full collapse keeps per-dimension spread at zero and rank at one while the loss sits at \(\log(2N-1)\). Partial collapse needs the same checks, since its loss still falls.
- SimSiam avoids collapse on positives alone with stop-gradient and a predictor. The constant solution still minimizes its loss; the paper shows that removing either piece collapses to 0.1 percent and offers an EM-style hypothesis, not a proof.
Reproduce closed-book
Traps
- Trusting falling loss as proof of learning. Partial collapse keeps the loss falling while dimensions die; only spread and rank checks catch it.
- Dismissing false negatives as rare. About 16 per anchor at SimCLR scale is a steady repulsive pressure on same-class views.
- Crediting the predictor alone for SimSiam. Figure 2 shows stop-gradient removal by itself collapses to 0.1 percent.
Practice
CS231n Assignment 3, Q2: augmentation ablation from scratch
Self_Supervised_Learning.ipynb: Data Augmentation (compute_train_transform, checked with rel_error), Train the SimCLR model, and the Finetune a Linear Layer comparison on the 10 percent split
The training cell loads pretrained weights and runs one epoch, so the ablation needs its own runs: skip the load_state_dict line, train a few epochs with and without color jitter at the notebook \(\tau\) of 0.5, then probe both. Small runs may show only a small gap; report what you measure, and run your collapse check on both embedding sets.
Self-test
-
What does the middle plot of SimSiam Figure 2 show without stop-gradient?
Per-channel std of the normalized output falls toward zero: the embeddings go constant and validation accuracy drops to 0.1 percent. -
Estimate same-class negatives per anchor at batch 8192 with 1000 balanced classes.
About 16: 16,382 negatives divided by 1000 classes. -
Your embeddings have numerical rank 3 in a 128-d space, but the loss keeps falling. Is that collapse?
Dimensional collapse: the loss can still fall inside the live subspace while most dimensions carry nothing. Report spread and the singular-value spectrum, and check the probe; a falling loss does not rule it out. -
Why does adding color distortion to cropping improve representations while making pairing harder?
It removes the color-histogram shortcut, so the encoder must use shape and content that transfer.
Completion criterion
Stop when
Your collapse check reports spread and numerical rank for constant, Gaussian and learned embeddings, and your ablation reports probe accuracy for both runs against the frozen random-encoder baseline.
Mark passed, then open 5.4 Only mark this when you can do it without notes.Leads to
- 5.4 Linear probes, transfer evaluation and pair leakage Check whether shortcut-laden representations survive a frozen probe and clean splits.
- 5.6 Paired modalities: projector heads, alignment and zero-shot retrieval See how paired text supervision changes the shortcut landscape.
Comes after
- 5.2 Temperature, negatives and positive constructionTest which augmentations create shortcuts and how collapse shows in the embeddings.
Resources for this unit
- Understanding Deep Learning - representation foundations Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
- SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
- SimSiam - Exploring Simple Siamese Representation Learning Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.
Also in this section
- D2L - Vision Transformer Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
- Stanford CS231n - transformer and self-supervised assignment slice Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
- TensorTonic - patchify and symmetric CLIP loss Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
- Deep-ML candidate - InfoNCE Loss If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
- CLIP - Learning Transferable Visual Models From Natural Language Supervision Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.