Unit 4.2 · about 4 h
194 / 324Pooling, transposed convolution, receptive fields and cost
Resolution sets both what a unit can see and what training costs. A receptive field smaller than the object and an activation bill larger than memory fail the same run.
- derive
You can now
- Derive receptive-field growth across stride, kernel and dilation choices.
- Count parameters, multiply-adds and activation bytes for a stated conv stack.
- Explain transposed convolution as learned upsampling and state when it is not an inverse.
Covers
- Pooling and transposed convolution at recognition depth
- Receptive fields, parameters, FLOPs and activation memory
Study, in this order
3 steps
- Pooling from scratchD2L - convolutional and recurrent architecture route
Section 7.5: Maximum and Average Pooling (
pool2dfrom scratch), Padding and Stride, and Multiple Channels - Transposed convolutionD2L - convolutional and recurrent architecture route
Sections 14.10.1 and 14.10.2: Basic Operation and Padding, Strides, and Multiple Channels, including Connection to Matrix Transposition
- Receptive fields, downsampling and upsamplingUnderstanding Deep Learning - convolution and residual networks
Chapter 10, Sections 10.2.6 and 10.4 (pp. 167 to 174): receptive fields with Figure 10.6, and downsampling plus upsampling with Figures 10.11 to 10.13
Depth · Skip Section 10.5 Applications. Detection and segmentation are later topics.
Focus on
- Receptive field grows by \((k-1)\) times the product of earlier strides per layer. Stride-2 downsampling doubles the growth rate while quartering activations.
- One conv layer holds \(C_{\text{out}}\) times \((C_{\text{in}}/G)\) times \(kH\) times \(kW\) weights plus \(C_{\text{out}}\) biases, and costs about twice that many FLOPs per output position for the multiply-add.
- Pooling has no parameters, acts per channel, and \(2 \times 2\) stride-2 pooling quarters resolution. Max-pooling gives limited shift tolerance inside a window, not invariance.
- A transposed convolution applies the transpose of a downsampling weight matrix. It learns an upsampling and is not an inverse of any forward convolution.
Reproduce closed-book
Conv2d(64, 128, kernel_size=3, stride=1, padding=1): output \(56 \times 56 \times 128\), 73,856 parameters, about 4.6e8 FLOPs, and about 1.6 MB of float32 output activations. Then grow the receptive field through two more \(k=3\) \(s=1\) layers to 7.Traps
- Quoting parameter counts as cost. FLOPs and activations scale with resolution, so doubling both sides quadruples both.
- Calling transposed convolution an inverse by name. It shares only the naming pattern, so its outputs need their own checks.
- Treating the receptive-field formula as uniform coverage. Center pixels feed more paths than edge pixels, so the formula gives the maximum extent.
Practice
UDL Chapter 10 Problems 10.8, 10.9, 10.11 and 10.14: channel counts and receptive fields
Problems (pp. 184 to 185): 10.8 (two-layer channel counts), 10.9 (three \(k=3\) layers), 10.11 (receptive field with a stride-2 layer) and 10.14 (\(5 \times 5\) kernel, 3 to 10 channels)
Solve on paper against the weight-shape formula before checking the section text.
TrenTorch: FLOPs Estimation (Linear/Conv)
Implement linear_flops and conv2d_flops with 2 FLOPs per multiply-add and one per bias add
Check it on the reproduce layer: Conv2d(64, 128, kernel_size=3, padding=1) on \(56 \times 56\) gives 462,422,016 FLOPs before bias plus 401,408 for the bias, about 4.6e8.
Self-test
-
Three \(k=3\) \(s=1\) layers on a 1D input. What is the receptive-field size?
7: \(1 + 3 \times 2\). Each \(k=3\) layer adds 2 to the extent at stride 1. -
Stride-2 pooling quarters the positions. Why does it not quarter the compute of the next stage?
Networks usually double the channels after downsampling. Convolution cost scales with positions times \(C_{\text{in}}\) times \(C_{\text{out}}\), so doubling both channel counts restores the FLOPs exactly, while activation memory only halves. -
Does a transposed convolution invert the convolution with the same weights?
No. It applies the transposed weight matrix with its own learned weights, so equal shape back does not mean equal values. -
\(2 \times 2\) stride-2 max-pooling. Which input shifts leave the output unchanged?
A shift by 2 pixels shifts the output by one cell (equivariance on the coarse grid). A one-pixel shift leaves an output unchanged only when that window's maximum stays inside the window, which depends on the data; test with a shifted copy.
Completion criterion
Stop when
Your hand counts match the weight-shape formula on a new layer, and a shifted-input probe shows exactly which pooling outputs move.
Mark passed, then open 4.3 Only mark this when you can do it without notes.Leads to
- 4.3 Equivariance, residual CNN blocks and resolution cost Test which of these mechanisms preserve or break translation equivariance.
- 5.5 Patch embeddings and positional information Compare receptive-field growth with patch tokenization of images.
Comes after
- 4.1 Convolution shapes: channels, padding, stride, dilation and groupsCarry shapes into pooling, learned upsampling and receptive-field growth.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
Also in this section
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.