Section 5 of 8 · Deep Learning

Representation, Self-Supervision & Paired Modalities

Representation objectives need an explicit invariance, pairing contract and held-out evaluator. A falling pretraining loss alone does not show useful transfer.

7 units 17 topics 0/7

Units

7 in this section

5.1 Embedding geometry and contrastive logits
derive
3 h
5.2 Temperature, negatives and positive construction
derive · measure
4 h
5.3 Augmentation shortcuts, false negatives and collapse
explain · measure
5 h
5.4 Linear probes, transfer evaluation and pair leakage
implement · measure
5 h
5.5 Patch embeddings and positional information
implement
3 h
5.6 Paired modalities: projector heads, alignment and zero-shot retrieval
explain · implement
4 h
5.7 Graph message passing and oversmoothing
explain
2 h

Before moving on

Checkpoint

Derive a contrastive objective, build a compact representation learner, detect a seeded pair leak, and defend held-out retrieval or transfer evidence with explicit limits.

Deliberately out of scope

Leave for later

Full graph neural network implementations
Large CLIP-style training pipelines
Multimodal generation and alignment research

Resources for this section

Shared by every unit above

Understanding Deep Learning - representation foundations
Simon J. D. Princetheory

primary

Use 9.3.6-9.3.7 for transfer and self-supervision, 12.9-12.10 for image transformers, and 13.1-13.4 plus 13.10 for GNN vocabulary. Open notebooks 9.4, 12.4 and 13.1 only when a figure or mechanism is unclear.
D2L - Vision Transformer
D2L authorsimplementation

required

Use vision-transformer.md for patch extraction, patch embeddings and a tested small encoder. Add a compact SimCLR-style or paired-data contrastive objective in the explicit training loop from Sections 2 and 3.
Stanford CS231n - transformer and self-supervised assignment slice
Stanford Universitycourse

support

Use only the Assignment 3 components for transformers, self-supervised learning, CLIP or DINO that directly test representation and pairing. Keep the controlled local evaluator as the gate.
TensorTonic - patchify and symmetric CLIP loss
TensorTonicpractice

support

Solve only Patchify a Batch of Images and Implement Symmetric CLIP Contrastive Loss. Transfer both into the controlled experiment and preserve local leakage and collapse tests.
Deep-ML candidate - InfoNCE Loss
Deep-MLpractice

optional

If the platform admission check passes, use problem 384 once after deriving the objective. Add local tests for temperature extremes, missing positives, duplicate pairs and normalized versus unnormalized embeddings.
SimCLR - A Simple Framework for Contrastive Learning of Visual Representations
Chen, Kornblith, Norouzi and Hinton (ICML 2020)paper

required

Use Section 2.1 Eq. (1) and Algorithm 1 for NT-Xent, 2.3 for the linear-evaluation protocol, 3.1 with Figures 5-6 for augmentation composition and shortcuts, 4.2 for the projection head, and 5.1-5.2 with Table 5 and Figure 9 for temperature and batch-size effects.
CLIP - Learning Transferable Visual Models From Natural Language Supervision
Radford et al. (OpenAI, ICML 2021)paper

required

Use Section 2.3 for the symmetric image-text objective and learned temperature, 2.5 for batch size and temperature limits, and 3.1.2 with 3.1.4 for zero-shot transfer and prompt ensembling.
SimSiam - Exploring Simple Siamese Representation Learning
Chen and He (CVPR 2021)paper

support

Use Figure 2 and Table 1 for collapse under removed stop-gradient or predictor, including the per-channel spread diagnostic.

Apply the section

Mini-project · Compact contrastive learner with a seeded leak audit (10 to 14 h)

Train a small SimCLR-style encoder on a CIFAR-10 subset, evaluate it with a frozen linear probe and held-out retrieval against a random-encoder baseline, and show that your split audit catches a pair leak you planted yourself.

Steps

Implement NT-Xent from your derivation. Pass Deep-ML 384 and the CS231n Assignment 3 simclr_loss_naive and simclr_loss_vectorized checks, and add tests for identical embeddings (\(\log(2N-1)\)), temperature extremes and unnormalized inputs.
Train a small encoder with a projector on a fixed CIFAR-10 subset at a stated temperature and batch size. Log loss, positive-pair accuracy and the collapse check (per-dimension spread of normalized embeddings and numerical rank) every epoch.
Evaluate with a frozen linear probe on \(h\) and on \(z\), and with nearest-neighbour retrieval on a held-out split, against a frozen random-encoder baseline with matched budgets, over two seeds.
Split by source image before augmenting, and write a split audit that combines exact hashes with a near-duplicate check, such as cosine similarity of downsampled pixels above a stated threshold.
Plant a leak: copy a few hundred test images, lightly augmented, into the training set. Show what the audit flags and misses, and measure how much the leak inflates retrieval and probe scores.
Run one ablation, such as dropping color jitter, and report probe accuracy and the collapse check for both runs.
Write the limits: what your subset, resolution and budget cannot show about large-scale pretraining or paired image-text alignment.

Deliverables

Training and evaluation code with the loss tests, the collapse check and the split audit.
A results table with seeds: probe on h and z, retrieval, the random-encoder baseline, the ablation, and leaked versus clean scores.
A one-page limits note.

Checks

external-tests · Deep-ML 384: Contrastive Loss (InfoNCE / SimCLR-style)
external-tests · CS231n Assignment 3, `Self_Supervised_Learning.ipynb` loss and augmentation checks
rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Objective	NT-Xent passes Deep-ML 384, the CS231n loss checks and your edge tests.	Your tests also cover duplicate images in one batch and show how they change the loss.
Evaluation protocol	The frozen probe and retrieval beat the random-encoder baseline at matched budgets.	The result holds on both seeds, and the gap between probing \(h\) and \(z\) is reported.
Leak detection	The audit flags the planted near-duplicates.	You report the audit's misses and false alarms on the planted set, and the score inflation the leak caused.
Collapse	Spread and rank are logged for every run.	A deliberately collapsed run, such as a constant projector output, trips the check.
Limits	The note names what the compact run cannot show.	Each limit names the measurement that would settle it.

Feeds into: Capstone artifact 3: the planted leak and its audit can be the objective or data postmortem.