Unit 3.7 · about 3 h
191 / 324Objective checks, tiny-set overfit and shuffled-label controls
A failing run is usually a broken objective before it is a broken optimizer. Shape, mask and reduction bugs change the loss silently, and the two controls separate them from optimization failures in minutes.
- implement
- measure
You can now
- Derive per-position cross-entropy with class weights,
ignore_indexmasks and mean versus sum reduction. - Run single-batch overfit and shuffled-label controls and state what each outcome proves about capacity, optimization and data.
Covers
- Objective target shapes, masks, weights and reductions
- Tiny-set overfit and shuffled-label controls
Study, in this order
4 steps
- Loss construction from likelihoodUnderstanding Deep Learning - fitting, gradients and regularization
Chapter 5: Maximum likelihood (5.1), Recipe for constructing loss functions (5.2) and multiclass classification (5.5)
- Target shapes, masks, weights and reductionsPyTorch CrossEntropyLoss reference
Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples - Random-label fitting evidenceUnderstanding Deep Learning - fitting, gradients and regularization
Section 8.4 with 15 percent randomized labels, and Chapter 20, Section 20.2.1: fitting random pixels and labels as capacity evidence
- Overfit, underfit and selectionD2L - optimization and regularization slice
Underfitting or Overfitting and Model Selection: capacity versus error and cross-validation discipline
Focus on
- Mean reduction divides by the number of non-ignored positions, or by the sum of their weights when
weightis set, not by the batch size. Averaging per-batch means with different valid counts weights tokens unequally; accumulate sums and counts instead. ignore_indexremoves padding from both loss and gradient but keeps the positions in the tensor. Weight rescales classes for imbalance; neither fixes mislabeled data.- Single-batch overfit that fails proves an optimization or capacity bug, not a data bug: the model cannot fit even a handful of examples.
- Shuffle the training labels only. On held-out data the shuffled run must score at chance; anything above chance means leakage between splits or an evaluation bug. On training data a large model can still memorize shuffled labels, more slowly than true ones, so a true-label run that trains no faster than the shuffled one is not using the signal.
Reproduce closed-book
Closed-book, compute cross-entropy for logits
[[2, 1, 0], [1, 2, 0]] with targets [0, -100] under ignore_index -100 and mean reduction: 0.408. Then write the two controls with their pass and fail readings.Traps
- Dividing a masked loss by the batch size or sequence length. PyTorch divides by the valid count (the sum of valid weights); a hand-written mask that sums and divides by the full size shrinks the loss and its gradient as padding grows.
- Expecting a shuffled-label run to fail to fit the training set. Overparameterized nets memorize random labels (UDL 20.2.1); the readings that matter are chance-level held-out accuracy and slower fitting than with true labels.
- Comparing cross-entropy values across different reductions or target sets as quality scores. Mean versus sum differ by the denominator, and different targets differ in what they average over.
Practice
PyTorch CrossEntropyLoss doc examples, extended
Examples with class indices and class probabilities; extend the first with ignore_index and weight and check the denominator by hand
Exercises: the zero-training-error question and the cross-validation questions
Self-test
-
Logits
[[2, 1, 0], [1, 2, 0]], targets[0, -100],ignore_index -100, mean reduction. What is the loss?The second position is ignored. First position: softmax gives \(p_0 = e^2/(e^2+e+1) = 0.665\), loss \(-\ln(0.665) = 0.408\). Mean over one valid position: 0.408. -
Single-batch overfit stalls at high loss. What is ruled out, and what remains?
Data scale, label noise and evaluation protocol are ruled out as the blocker. What remains is capacity, objective wiring, or optimization: learning rate, initialization or gradient flow. -
Shuffled-label training reaches near-zero loss but true-label training does not generalize. What does the control prove?
It proves the model and optimizer can fit arbitrary patterns, so capacity and optimization work. The true-label failure is about signal: features, labels or leakage, not fitting power. -
A shuffled-label run scores 30 percent held-out accuracy on a balanced 10-class task. What does that mean?
Chance is 10 percent, so something leaks: duplicates across splits, labels recoverable from features such as file names or order, or an evaluation bug. Find it before trusting any true-label number. -
Class weight 5.0 on the rare class under mean reduction. How does one rare example count?
Its unreduced loss is multiplied by 5 and the mean divides by the sum of valid weights, so it pulls the average five times harder than an unweighted example.
Completion criterion
Stop when
Your hand-computed masked loss matches the framework to three decimals, and both controls give the predicted reading on a broken and a healthy run.
Mark passed, then open 3.8 Only mark this when you can do it without notes.Leads to
- 3.8 Diagnosing failed runs: divergence, stale state, slices and seeds Turn control failures into ordered repair drills.
- 1.2 Stable softmax, cross-entropy and loss reductions Ground the loss in stable softmax and reduction semantics.
Comes after
- 1.5 Gradient checks with finite differencesExtend checks to tiny-set overfit and shuffled-label controls.
- 2.7 Explicit training and evaluation loops with correct metricsAdd overfit and shuffled-label controls to the loop.
- 3.4 Regularization: dropout, label smoothing, augmentation and weight decayLabel smoothing changes the loss targets; check them there.
- 3.6 Schedules, gradient accumulation and clippingCheck the objective the schedule is optimizing before tuning further.
- 5.4 Linear probes, transfer evaluation and pair leakageReuse tiny-set overfit and shuffled-label controls to test your probe pipeline.
- 8.9 Profiling training phases and verifying optimizationsReuse the overfit and shuffled-label controls as learning checks.
Resources for this unit
- Understanding Deep Learning - fitting, gradients and regularization Use Chapters 6-9 and targeted parts of Chapter 20 for optimization, initialization, measurement, regularization and why training works. Re-derive the update rules you keep in the trainer.
- D2L - optimization and regularization slice Use MLP implementation, dropout, deep generalization, numerical stability, optimization introduction, SGD, minibatch SGD, momentum, AdaGrad, RMSProp, Adam and learning-rate scheduler. Implement momentum, AdamW, initialization, dropout, one normalization, clipping and one schedule.
- PyTorch CrossEntropyLoss reference Read Shape, Parameters (
weight,ignore_index,reduction,label_smoothing) and both Examples; extend the class-index example withignore_indexand a classweight.
Also in this section
- PyTorch - optimization and automatic mixed precision examples Check optimizer, scheduler, gradient clipping and AMP API semantics. Preserve the correct order for zeroing, scaling, backward, unscaling, clipping, stepping and updating.
- Deep-ML candidate - optimization and normalization problems After the platform admission check, use at most one of 146 Momentum, 169 AdamW, 197 Global-Norm Clipping, 109 LayerNorm or 115 BatchNorm as a changed-prompt drill.
- TrenTorch - training mechanics questions Solve only the questions the units name: Weight Initialization, Layer Normalization, Label Smoothing, Gradient Accumulation, and Warmup and Cosine Decay. Read the tests first; they run in the browser IDE.
- PyTorch GroupNorm reference Read how input channels are split into
num_groupsgroups and which axes each group normalizes; comparenum_groups1 againstnum_channels. - PyTorch RMSNorm reference Read the RMS formula and which axes
normalized_shapecovers; note the missing mean centering and bias. - PyTorch torch.optim.AdamW reference Read the AdamW algorithm box for the decoupled weight-decay step, the bias-corrected moments and the defaults, and follow the Decoupled Weight Decay Regularization link for the argument.
- Cyclical Learning Rates for Training Neural Networks (Smith, 2017) Read the abstract and the range test: increase the rate linearly for a few epochs and read usable bounds off the loss curve.
- Deep Learning Tuning Playbook FAQ: debugging optimization failures Read How can optimization failures be debugged and mitigated (identifying unstable workloads, warmup, clipping) and Batch normalization implementation details.
- Hands-On Machine Learning with Scikit-Learn and PyTorch Read the Chapter 11 sections named in the units for the PyTorch side of initialization, normalization, clipping and schedules.