Unit 4.1 · about 5 h
193 / 324Convolution shapes: channels, padding, stride, dilation and groups
Every CNN bug report starts with a shape mismatch. Channels, padding and groups decide tensor layout before any learning happens.
- derive
- implement
You can now
- Derive output shapes with padding, stride, dilation and groups, including the effective kernel size \(d(k-1)+1\).
- State the weight layout \((C_{\text{out}}, C_{\text{in}}/G, kH, kW)\) and count parameters for plain, grouped and depthwise cases.
- Implement cross-correlation from loops and match a framework
Conv2don shapes and values.
Covers
- Convolution kernels, channels and output shapes
- Padding, stride, dilation and groups
Study, in this order
5 steps
- Cross-correlation, kernels and the feature mapD2L - convolutional and recurrent architecture route
Section 7.2: The Cross-Correlation Operation (
corr2dfrom scratch), Convolutional Layers (theConv2Dclass), Object Edge Detection in Images, Learning a Kernel, and Cross-Correlation and Convolution - Padding and stride shapesD2L - convolutional and recurrent architecture route
Sections 7.3.1 and 7.3.2: the padding formula (7.3.1), the stride formula (7.3.2), and the
comp_conv2dshape demos including kernel \((3, 5)\) with padding \((0, 1)\) and stride \((3, 4)\) - Channels and \(1 \times 1\) convolutionsD2L - convolutional and recurrent architecture route
Section 7.4: multiple input channels, multiple output channels, and \(1 \times 1\) convolutions with the reshape-plus-matmul check
Depth · Skip the ResNeXt preview at the end. Grouped cost returns through the PyTorch reference below.
- Stride, dilation and channelsUnderstanding Deep Learning - convolution and residual networks
Chapter 10, Sections 10.2.3 to 10.2.5 (pp. 164 to 165): stride, kernel size, dilation rate as zeros-plus-one, and channel counts \(Ci\) by \(Co\) by \(K\)
- The exact shape and groups contractPyTorch nn.Conv2d reference
Signature and Shape: the floor formula with dilation, the groups divisibility rule with the
groups=2and depthwise examples, and the Variables weight shape
Focus on
- Deep-learning convolution is cross-correlation with no kernel flip. A learned kernel absorbs the flip, so the distinction changes nothing in a trained layer.
- Output size per axis is \(\left\lfloor (n + 2p - d(k-1) - 1)/s + 1 \right\rfloor\). With odd \(k\) and matching same padding each output centers on its input pixel.
- Groups split channels into \(G\) independent convolutions. Parameters and multiply-adds fall by \(G\), while output shape is unchanged. Groups equal to \(C_{\text{in}}\) is depthwise convolution.
- A \(1 \times 1\) convolution is a per-pixel linear map across channels. Stacked \(1 \times 1\) layers need a nonlinearity between them or they fold into one linear map.
Reproduce closed-book
Conv2d(3, 64, kernel_size=7, stride=2, padding=3) on \(224 \times 224\) gives \(112 \times 112\) with 9,472 parameters. Then implement corr2d from loops and match torch.nn.Conv2d values on a \(5 \times 5\) input.Traps
- Counting padding once in the formula. D2L and PyTorch \(p\) means padding on both sides, so the total added rows are \(2p\).
- Assuming groups change the output shape. Groups change only the weight layout and the cost, not \(H_{\text{out}}\) or \(W_{\text{out}}\).
- Reading convolution as flipped correlation and hand-flipping a learned kernel before reuse. Export the kernel as is.
Practice
CS231n Assignment 2 Q3: convolution layers in `ConvolutionalNetworks.ipynb`
Q3: Convolutional Neural Networks, ConvolutionalNetworks.ipynb in the starter code (https://cs231n.github.io/assignments/2026/assignment2.zip): Convolution: Naive Forward Pass and Naive Backward Pass. Each cell compares your output with expected values or numerical gradients through rel_error.
Gradescope is for enrolled students; the notebook checks are the gate. Expect relative errors around 1e-8 or smaller.
Self-test
-
\(3 \times 3\) input, \(2 \times 2\) kernel, no padding, stride 1. What is the output shape and why?
\(2 \times 2\). The 2-wide window fits \(3 - 2 + 1 = 2\) times per axis; without padding a \(k\)-wide kernel loses \(k - 1\) positions per axis. -
Conv2d(32, 32, kernel_size=3, groups=8). What is the weight shape and the parameter count with bias?\((32, 4, 3, 3)\): \(32 \times 4 \times 9 = 1{,}152\) weights plus 32 biases, or 1,184. The dense version needs 9,216 weights plus 32 biases. -
Kernel size 3 with dilation 2. What is the effective kernel size and what does it buy?
5, from \(d(k-1)+1\). A \(3 \times 3\) kernel with dilation 2 sees a \(5 \times 5\) window with 9 weights per channel pair, at the multiply-add cost of an ordinary \(3 \times 3\) kernel. -
Two stacked \(1 \times 1\) convolutions with no nonlinearity between them. What is the problem?
They compose into a single linear map. Keep a \(1 \times 1\) layer only with a nonlinearity or normalization separating it from the next linear map.
Completion criterion
Stop when
Your loop corr2d matches Conv2d values, and you derive grouped and dilated shapes without notes.
Leads to
- 4.2 Pooling, transposed convolution, receptive fields and cost Carry shapes into pooling, learned upsampling and receptive-field growth.
- 4.3 Equivariance, residual CNN blocks and resolution cost Use same-shape padding to reason about equivariance and residual adds.
Resources for this unit
- D2L - convolutional and recurrent architecture route Use why convolution, convolution layer, padding and strides, pooling, channels, batch normalization and ResNet, then RNN, BPTT, scratch RNN, GRU, LSTM, encoder-decoder and seq2seq. Work through the scratch mechanisms before concise APIs.
- Understanding Deep Learning - convolution and residual networks Use Chapters 10 and 11 to sharpen convolutional invariance, receptive-field reasoning, residual paths and normalization. D2L remains the recurrent implementation route.
- PyTorch nn.Conv2d reference Read Signature and Shape for the exact floor formula with dilation, the groups divisibility rule with grouped and depthwise examples, and the weight layout \((C_{\text{out}}, C_{\text{in}}/\text{groups}, kH, kW)\). D2L and UDL teach the mechanism; this page is the shape contract to code against.
Also in this section
- Stanford CS231n - Assignments 1 and 2 Use only the neural network, normalization, dropout, convolution and PyTorch components that match this section. Do not complete the course or unrelated vision tasks.
- Deep-ML candidate - convolution and recurrence problems After the platform admission check, use at most one of 41 Conv2D, 62 RNN with BPTT, 287 GRU or 59 LSTM as a changed-prompt drill.
- TrenTorch - convolution cost and recurrent cells Solve FLOPs Estimation, GRU Forward and LSTM Forward after the D2L scratch models. Each reference cell matches torch.nn.GRUCell or LSTMCell.
- PyTorch sequence packing utilities Read
pack_padded_sequenceforlengths,enforce_sortedand batch-first at recognition depth: what packing skips and what contract it demands. Do not reimplement the packed format; pair it with the masked-loss implementation in D2L seq2seq.