Unit 5.4 · about 5 h
203 / 324Linear probes, transfer evaluation and pair leakage
A probe is the representation's external exam. A leaked pair turns the exam into a memory test, so the split discipline matters as much as the accuracy.
- implement
- measure
You can now
- Implement a frozen-encoder linear probe with matched baseline budgets.
- Evaluate transfer with held-out retrieval or classification and state slice limits.
- Build pair-identity and near-duplicate checks that catch train/test leakage.
Covers
- Frozen encoders, linear probes and fine-tuning
- Held-out retrieval and transfer evaluation
- Pair, identity and near-duplicate leakage
Study, in this order
4 steps
- Linear evaluation protocolSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 2.3: linear classifier on the frozen base network with test accuracy as the representation-quality proxy
- Transfer learning and fine-tuningUnderstanding Deep Learning - representation foundations
Section 9.3.6 (pp. 151-152) with Figure 9.12a (p. 153): pretrain on a secondary task, replace the last layers, then train only the new layers or fine-tune all
- Probe the encoder, not the projectorSimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Section 4.2 with Table 3: \(h\) beats \(z\) by over 10 points; rotation prediction 67.6 percent from \(h\) versus 25.6 percent from \(g(h)\)
- Frozen linear probe on CIFARStanford CS231n - transformer and self-supervised assignment slice
Self_Supervised_Learning.ipynb: Classifier with frozenModel().f, Finetune a Linear Layer for Classification, Baseline Without Self-Supervised Learning versus With Self-Supervised Learning, Plot your Comparison
Focus on
- Freeze the encoder (
requires_gradFalse) and train only the head. Anything else measures supervised relearning. - Probe \(h\), the pre-projection features. In SimCLR Table 3, where rotation was added to the augmentations, the projector output \(z\) lost the rotation information that \(h\) kept.
- Match budgets exactly: same data fraction, epochs and head settings for baseline and pretrained runs.
- Split by source identity before augmenting. Two views of one image on opposite sides of the split leak the answer.
Reproduce closed-book
Traps
- Probing \(z\) instead of \(h\). In SimCLR Table 3 an MLP on \(h\) predicts rotation at 67.6 percent and one on \(z\) at 25.6, near the 25 percent chance level, because rotation was one of the augmentations.
- Celebrating a probe win with mismatched budgets. A longer-trained or larger-split baseline can erase the gap.
- Splitting augmented views independently. Near-duplicates across the split inflate retrieval and probe scores without teaching anything.
Practice
CS231n Assignment 3, Q2: SimCLR plus linear probe
Self_Supervised_Learning.ipynb: simclr_loss_naive and simclr_loss_vectorized in cs231n/simclr/contrastive_loss.py, Train the SimCLR model, then Finetune a Linear Layer for Classification with the frozen encoder.
Match the naive and vectorized losses first; the probe comparison is only meaningful on a correct objective. The section mini-project adds seeded pair-leak detection and a held-out retrieval defense.
Self-test
-
Why does the notebook set
requires_gradFalseonmodel.f?To freeze the encoder so test accuracy measures representation quality, following the SimCLR Section 2.3 protocol. -
Rotation prediction: \(h\) scores 67.6 percent, \(g(h)\) 25.6 percent. What does that tell you about probing?
Probe \(h\). In that run rotation was one of the augmentations, so the contrastive loss trained \(z\) to ignore it; probing \(z\) understates what the encoder keeps. -
Your retrieval score rises after adding test pairs mined from the training distribution. Leak or gain?
Suspect near-duplicate leakage first. Re-split by pair identity, re-run, and only then claim a gain. -
Your probe reaches a reference accuracy quoted in a notebook comment, but your baseline trained for fewer epochs. What can you claim?
Nothing about the representation yet. Only a comparison with matched data fraction, epochs and head settings isolates the effect of pretraining.
Completion criterion
Stop when
Your frozen probe beats the matched no-pretrain baseline, your naive and vectorized losses agree, and your split audit shows no shared source identities across train and test.
Mark passed, then open 5.5 Only mark this when you can do it without notes.Leads to
- 5.6 Paired modalities: projector heads, alignment and zero-shot retrieval Evaluate paired representations with retrieval and zero-shot probes.
- 3.7 Objective checks, tiny-set overfit and shuffled-label controls Reuse tiny-set overfit and shuffled-label controls to test your probe pipeline.
Comes after
- 5.3 Augmentation shortcuts, false negatives and collapseCheck whether shortcut-laden representations survive a frozen probe and clean splits.
- 5.6 Paired modalities: projector heads, alignment and zero-shot retrievalJudge alignment with frozen probes and leak-checked retrieval splits.
Resources for this unit
- Understanding Deep Learning - representation foundations Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
- Stanford CS231n - transformer and self-supervised assignment slice Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
- SimCLR - A Simple Framework for Contrastive Learning of Visual Representations Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
Also in this section
- D2L - Vision Transformer Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
- TensorTonic - patchify and symmetric CLIP loss Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
- Deep-ML candidate - InfoNCE Loss If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
- CLIP - Learning Transferable Visual Models From Natural Language Supervision Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.
- SimSiam - Exploring Simple Siamese Representation Learning Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.