Unit 4.3 · about 5 h
195 / 324Equivariance, residual CNN blocks and resolution cost
Residual structure and augmentation decide whether a deeper or larger-input model trains at all. Equivariance claims decide whether it should have been a CNN in the first place.
- explain
- measure
You can now
- Explain what translation equivariance promises and name three mechanisms that break it: stride, pooling and padding boundaries.
- Explain the residual unraveled-path view and why identity paths stabilize deep CNNs.
- Measure resolution-driven time and memory across two input sizes and attribute the change.
Covers
- Translation equivariance and its limits
- Residual blocks, normalization and initialization in CNNs
- Augmentation and resolution-driven cost
Study, in this order
4 steps
- Why convolution: invariance and localityD2L - convolutional and recurrent architecture route
Sections 7.1.1 to 7.1.4: Invariance, Constraining the MLP with translation invariance and locality, and Convolutions
Depth · Skim the Channels section. Channel mechanics belong to the first unit.
- Residual blocks and ResNetD2L - convolutional and recurrent architecture route
Sections 8.6.1 to 8.6.3: Function Classes with nested sets, Residual Blocks, and the ResNet Model stem plus four stages
Depth · Skim ResNeXt and Training. You train your own small net in practice below.
- Equivariance, residual paths and normalizationUnderstanding Deep Learning - convolution and residual networks
Chapter 10, Section 10.1 (p. 161) and Chapter 11, Sections 11.2 to 11.4 (pp. 189 to 193): equivariance versus invariance, residual blocks and operation order, exploding variance and batch normalization
Depth · Skip Section 11.5 architectures and Section 10.5 applications. DenseNet, U-Net, detection and segmentation are later topics.
- Image augmentation methodsD2L - convolutional and recurrent architecture route
Section 14.1: Common Image Augmentation Methods with flipping, cropping and color changes
Focus on
- Convolution is equivariant to translation in the interior. Stride, pooling and zero-padded borders break exact equivariance, and pooling trades it for local tolerance.
- Zero padding leaks absolute position. The network can learn where the whitespace border is.
- A residual block fits the difference from identity, so pushing branch weights toward zero recovers a shallower net. The unraveled sum gives \(2^K\) paths for \(K\) blocks.
- Each residual addition grows activation variance under standard initialization. Per-channel batch normalization, or branch rescaling, keeps the forward stable.
Reproduce closed-book
Traps
- Calling CNNs translation invariant. Convolutions are approximately equivariant; invariance comes from pooling, above all global pooling before the classifier.
- Leaving batch normalization in training mode at test time. Small or single-example batches then rescale outputs with batch noise.
- Reading higher resolution as free accuracy. Activations and per-layer FLOPs grow about fourfold when both sides double, so measure before committing.
Practice
CS231n Assignment 2 Q4: train a convnet on CIFAR-10 in PyTorch
Q4: PyTorch on CIFAR-10, PyTorch.ipynb in the starter code (https://cs231n.github.io/assignments/2026/assignment2.zip): design and train a convolutional network on CIFAR-10.
Gradescope is for enrolled students; your own held-out accuracy is the gate. Log time and peak memory at two input resolutions and attribute the change before tuning accuracy.
CS231n Assignment 2 Q1: batch and layer normalization
Q1: Batch Normalization, BatchNormalization.ipynb in the starter code (https://cs231n.github.io/assignments/2026/assignment2.zip): batch and layer normalization forward and backward, checked with rel_error against numerical gradients.
Self-test
-
Stride-2 convolution on an input shifted by one pixel. Does the output shift by half a pixel?
No. Outputs live on the subsampled grid, so a one-pixel input shift can change which inputs each output sees. Exact equivariance fails under stride. -
Why can a deeper ResNet train when a deeper plain net degrades?
Each block can push its branch to zero and pass identity through, so the deeper net contains the shallower one. Plain stacks must fit identity through nonlinear layers. -
Batch normalization left in training mode at test time with batch size 1. What breaks?
Statistics come from that one example (over its spatial positions in a conv layer), so the layer acts like instance normalization: outputs depend on the example's own contrast instead of the training distribution, and the running buffers get overwritten with test data. Useevalmode. -
Random cropping as augmentation. What is the failure mode to guard against?
A crop that removes the object teaches guessing from background. Keep crops large relative to the object and compare against a no-augmentation baseline.
Completion criterion
Stop when
You can break and repair equivariance with shift probes, and a two-resolution timing run attributes time and memory changes to resolution.
Mark passed, then open 4.4 Only mark this when you can do it without notes.Leads to
- 3.2 Dead, saturated and exploding units, and residual paths Place residual paths beside dead units and normalization-axis choices.
- 7.7 Architecture trade-offs and attention correctness tests Compare convolutional inductive bias against attention trade-offs.
Comes after
- 4.1 Convolution shapes: channels, padding, stride, dilation and groupsUse same-shape padding to reason about equivariance and residual adds.
- 4.2 Pooling, transposed convolution, receptive fields and costTest which of these mechanisms preserve or break translation equivariance.
- 5.7 Graph message passing and oversmoothingCompare graph permutation equivariance with translation equivariance, and residual paths that allow depth.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
Also in this section
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.