22 Convolutional Neural Networks

“The face is not a simple pattern; it is a hierarchy of parts within parts within parts.”—David Marr, Vision (1982)

The feed-forward networks of Chapter 18 treat every input dimension identically. When the input is an image, this ignores spatial structure: nearby pixels are more related than distant ones, and the same pattern (an edge, a texture) can appear anywhere in the frame. Convolutional neural networks encode these two priors, locality and translation equivariance, directly into the architecture.

The foundational architecture for convolutional neural networks emerged in 1980, when Kunihiko Fukushima introduced the Neocognitron (Fukushima 1980), a hierarchical neural network inspired by Hubel and Wiesel’s discoveries about simple and complex cells in the mammalian visual cortex. The Neocognitron was the first architecture to use convolutional layers for feature extraction and pooling layers for translation invariance, successfully recognizing simple patterns through unsupervised learning, regardless of their position in the image. In 1988, Wei Zhang and colleagues applied backpropagation to train a simplified Neocognitron for alphabet recognition (Zhang et al. 1988), demonstrating that supervised learning could be effectively combined with the convolutional architecture. This combination of Fukushima’s architectural insights with gradient-based supervised learning became the foundation for modern CNNs, which now power everything from facial recognition on smartphones to autonomous vehicle perception systems.

We first calculate a filter response by hand, then combine two filters into a classifier. The MNIST example then learns filters from data and compares a CNN with the fully connected networks of Chapter 18.

22.1 Convolutions

A filter (also called a kernel) is a small matrix of weights, typically \(3\times3\) or \(5\times5\), that extracts local features by computing a weighted sum of pixel values within a sliding window. The same weights are used at every position.

Key Concepts

Consider the upper-left and upper-right \(3\times3\) patches of a \(5\times5\) cross formed by its two diagonals. Call these patches \(P\) and \(Q\), and use a filter \(f\) with ones along its main diagonal: \[ P=\begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix},\qquad Q=\begin{pmatrix}0&0&1\\0&1&0\\1&0&0\end{pmatrix},\qquad f=\begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}. \] Multiply corresponding entries and add. With rows and columns indexed from zero, the responses at positions \((0,0)\) and \((0,2)\) are \[ \begin{aligned} C_{0,0}&=1\cdot1+0\cdot0+0\cdot0+0\cdot0+1\cdot1+0\cdot0+0\cdot0+0\cdot0+1\cdot1=3,\\ C_{0,2}&=0\cdot1+0\cdot0+1\cdot0+0\cdot0+1\cdot1+0\cdot0+1\cdot0+0\cdot0+0\cdot1=1. \end{aligned} \] The matching diagonal gives the larger response; the opposite diagonal still shares the center pixel. Repeating this calculation across the image produces a feature map recording where the filter responds.

Now place a shorter diagonal stroke on a blank image and shift it one pixel right, without clipping any of its pixels: \[ X=\begin{pmatrix}1&0&0&0&0\\0&1&0&0&0\\0&0&1&0&0\\0&0&0&0&0\\0&0&0&0&0\end{pmatrix} \quad\longrightarrow\quad X'=\begin{pmatrix}0&1&0&0&0\\0&0&1&0&0\\0&0&0&1&0\\0&0&0&0&0\\0&0&0&0&0\end{pmatrix}. \] The response of 3 moves from \((0,0)\) to \((0,1)\). This is translation equivariance: for a shared filter with stride one, moving the pattern moves its response, away from boundary effects. Invariance would mean an unchanged output. A maximum over the whole feature map is 3 for either image, but no longer tells us where the stroke was.

The stride is the number of pixels by which the filter shifts at each step. A stride of 1 moves pixel by pixel; a stride of 2 skips positions. The receptive field is the input region influencing one output value, here \(3\times3\). Padding adds zeros around the border: with stride one, “same” padding preserves the input dimensions, while “valid” padding uses no padding and shrinks the output.

A related local weighted sum is kernel smoothing. For a one-dimensional signal \(x\) and a filter \(w\) of length \(h\), the unflipped operation is \[ C(t)=\sum_{i=0}^{h-1}x(t+i)w(i). \] Strictly, this is cross-correlation: convolution in signal processing flips the filter and sums \(x(t-i)w(i)\). Deep learning libraries omit the flip and still call the operation convolution; learned weights can absorb that reversal. The toy below also uses this unflipped convention. For a symmetric filter such as the moving average below, the two agree up to a shift of the output index.

In Figure 22.1, a uniform filter of length 10, with each weight \(1/10\), smooths a noisy sine wave over \([0,10]\). Smoothing uses fixed averaging weights; learned image filters can instead emphasize changes such as edges.

Figure 22.1: Smoothing a noisy signal using 1D convolution with a moving average filter. The light gray line shows the original noisy sinusoidal signal, while the black line displays the smoothed output after applying a uniform filter of size 10.

A Toy Example

To build intuition, consider a simple example with \(5\times5\) images and \(3\times3\) filters. We will classify images into three classes: cross, diagonal, and right-diagonal.

We will use two filters.

The first filter has 1s along its main diagonal and 0s elsewhere, as in our hand calculation. The second detects the opposite diagonal. The toy network takes a few lines of code: conv slides a \(3\times3\) filter over the image, maxpool performs global max pooling by keeping the strongest response over the entire feature map, and fc followed by softmax turns the two pooled responses into class probabilities. Both the filters and classifier weights w are set by hand, not learned.

def conv(img, f):
    out = np.zeros((3,3))
    for i in range(3):
        for j in range(3):
            out[i,j] = np.sum(img[i:i+3,j:j+3]*f)
    return out
def maxpool(img):
    return np.max(img)
def fc(x, w, b):
    return jnp.dot(w,x) + b
def softmax(x):
    return jnp.exp(x) / jnp.sum(jnp.exp(x))

def cnn(img, f1, f2):
    out1 = conv(img,f1)
    out2 = conv(img,f2)
    x = np.array([maxpool(out1), maxpool(out2)])
    w = np.array([[1,0],[0,1],[0.6,0.6]])
    b = 0
    z = fc(x,w,b)
    return softmax(z),out1,out2

Both feature maps for the cross have a maximum of 3. Applying the same two filters to the diagonal image gives different responses.

The opposite-diagonal filter produces the weaker map \(\left(\begin{smallmatrix}1&0&1\\0&1&0\\1&0&1\end{smallmatrix}\right)\), rather than a zero response. Global max pooling reduces each map to one number:

Image	Filter 1 maximum	Filter 2 maximum
Diagonal	3	1
Right-diagonal	1	3
Cross	3	3

For the diagonal image, the classifier multiplies \((3,1)\) by its three weight rows to obtain logits \((3,1,2.4)\). The cross row \([0.6,0.6]\) rewards both diagonals together: for \((3,3)\) it gives 3.6, exceeding either individual response of 3. Softmax converts these logits to probabilities.

The three class probabilities, ordered diagonal, right-diagonal, cross, are (0.26, 0.26, 0.48) for the cross image, (0.59, 0.08, 0.33) for the diagonal image and (0.08, 0.59, 0.33) for the right-diagonal image. The model correctly predicted all three classes. Global pooling discards position, as in the shifted-stroke example, but also loses spatial relationships that may matter for classification.

We now learn both filters and classifier weights from labeled images.

22.2 The MNIST Dataset

The MNIST dataset contains grayscale images of handwritten digits, each \(28\times28\) pixels, labeled 0 through 9. The original dataset includes 60,000 training images and 10,000 test images. For simplicity, both examples below use only the official test data, shuffled into 8,000 training images and 2,000 held-out evaluation images. We call the latter the held-out set throughout; this is not the standard MNIST training/test protocol.

Image data are represented as a matrix of pixel values. A grayscale image has shape (28, 28), with intensities from 0 (black) to 255 (white). Color images add red, green, and blue channels, giving shape (28, 28, 3). Figure 22.2 shows sample digits.

22.3 Fully Connected Approach

A natural starting point is a fully connected neural network, which reshapes each image into a vector of 784 values with no built-in notion of neighboring positions. Its units combine pixels, but separate weights must learn the same pattern at different locations. The output contains probabilities for the 10 classes.

The JAX implementation follows Chapter 18 and the gradient updates of Chapter 20. Pixel values are scaled to \([0,1]\), and labels are stored as one-hot vectors.

The cross-entropy loss selects the predicted log-probability of the true digit; accuracy compares the most probable class with the label.

from jax import jit, grad
import jax.numpy as jnp
from jax.scipy.special import logsumexp

def loss(params, batch):
  inputs, targets = batch
  preds = predict(params, inputs)
  return -jnp.mean(jnp.sum(preds * targets, axis=1))

def accuracy(params, batch):
  inputs, targets = batch
  target_class = jnp.argmax(targets, axis=1)
  predicted_class = jnp.argmax(predict(params, inputs), axis=1)
  return jnp.mean(predicted_class == target_class)

The network has two hidden layers with \(\tanh\) activation. Its final layer returns log-probabilities, computed as \(z_i-\log\sum_j e^{z_j}\) using logsumexp for numerical stability.

def predict(params, inputs):
    activations = inputs
    for w, b in params[:-1]:
        outputs = jnp.dot(activations, w) + b
        activations = jnp.tanh(outputs)

    final_w, final_b = params[-1]
    logits = jnp.dot(activations, final_w) + final_b
    return logits - logsumexp(logits, axis=1, keepdims=True)

We initialize the weights, form mini-batches of 128 images, and use SGD with step size 0.001 for 10 epochs.

layer_sizes = [784, 1024, 1024, 10]
param_scale = 0.1
step_size = 0.001
num_epochs = 10
batch_size = 128

num_train = train_images.shape[0]
num_complete_batches, leftover = divmod(num_train, batch_size)
num_batches = num_complete_batches + bool(leftover)

def init_random_params(scale, layer_sizes, rng=npr.RandomState(0)):
  return [(scale * rng.randn(m, n), scale * rng.randn(n))
          for m, n, in zip(layer_sizes[:-1], layer_sizes[1:])]

def data_stream():
    while True:
        perm = rng.permutation(num_train)
        for i in range(num_batches):
            batch_idx = perm[i * batch_size:(i + 1) * batch_size]
            yield train_images[batch_idx], train_labels[batch_idx]
batches = data_stream()

@jit
def update(params, batch):
    grads = grad(loss)(params, batch)
    return [(w - step_size * dw, b - step_size * db) for (w, b), (dw, db) in zip(params, grads)]

The architecture is shown in Figure 22.3. The run achieves 80% training accuracy and 77% held-out accuracy; this run also leaves substantial training error.

Fully connected network for MNIST with a 784-node input layer, two hidden layers of 1024 tanh nodes and a 10-node log-softmax output layer that gives the predicted digit, with the weight and bias shapes written on each connection. — Figure 22.3: Fully Connected MNIST

Figure 22.4 shows the training and held-out accuracy during training.

Figure 22.4: Accuracy of the fully connected network on the held-out and training sets.

A fully connected layer connecting a \(28\times28\) image to 1024 hidden units requires over 800,000 parameters. A shared \(3\times3\) filter uses nine weights at every position, avoiding separate copies of a detector for different locations.

22.4 CNN for MNIST

We use PyTorch’s nn.Conv2d to implement learned filters. Each first-layer filter produces one feature map. A second-layer filter spans all 32 input maps, sums their local weighted responses, and adds a bias to produce one new map. Thus each second-layer output channel has \(32\times3\times3+1=289\) parameters, and its 64 channels have 18,496 parameters. Two stride-one \(3\times3\) layers have a \(5\times5\) receptive field before pooling: the second layer combines features from neighboring first-layer windows.

The shapes in Figure 22.5 follow \(1\times28\times28\), \(32\times26\times26\), \(64\times24\times24\), then \(64\times12\times12\) after local \(2\times2\) max pooling. Unlike the toy’s global maximum, local pooling retains a grid of responses. It can reduce sensitivity to some shifts, but does not guarantee invariance; the flattened dense head also uses position.

Architecture of the convolutional network: a 1 by 28 by 28 grayscale input, two 3 by 3 convolutions with ReLU, max pooling and flattening to 9216 features, then a 9216 to 128 linear layer with ReLU and a 128 to 10 linear layer with log-softmax giving 10 digit classes. — Figure 22.5: CNN Architecture

import torch
import torch.nn as nn
from torch.nn import functional as F
x = parse_images("data/t10k-images-idx3-ubyte.gz")
y = parse_labels("data/t10k-labels-idx1-ubyte.gz")
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1, 32, 3, 1)  # 1 input channel (grayscale), 32 output channels, 3x3 kernel
        self.conv2 = nn.Conv2d(32, 64, 3, 1)  # 32 input channels, 64 output channels
        # Note: PyTorch defaults to valid padding (no padding). 
        # After conv1 (28->26), conv2 (26->24), pool (24->12): 64 channels × 12 × 12 = 9216
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)
    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

Training draws the same 8,000 images as the dense network, as tensors of shape (batch, 1, 28, 28). The second axis is the grayscale channel expected by nn.Conv2d.

def train(model, optimizer):
    model.train()
    batches = data_stream()
    for epoch in range(num_epochs):
        for _ in range(num_batches):
            optimizer.zero_grad()
            data, target = next(batches)
            output = model(data)
            loss = F.nll_loss(output, target)
            loss.backward()
            optimizer.step()
        print(f"Epoch {epoch}: Loss {loss.item()}")

torch.manual_seed(0)
model = Net()
optimizer = torch.optim.Adadelta(model.parameters(), lr=0.1)
train(model, optimizer)

This run uses Adadelta with learning rate 0.1, rather than the dense network’s SGD. We evaluate on the same 2,000 held-out images.

The two implementations give the following reported results. Both use 10 epochs and mini-batches of 128; parameter counts include biases.

Model	Training setup	Held-out accuracy	Parameters
Fully connected	JAX, \(\tanh\), SGD, step size 0.001	77%	1,863,690
CNN	PyTorch, ReLU, Adadelta, learning rate 0.1	93%	1,199,882

The accuracy gap does not isolate convolution: architecture, activation, optimizer, and framework all change. These are two illustrative implementations, not evidence that convolution alone raises accuracy from 77% to 93%. A controlled comparison would use matched training recipes and evaluate both models on the same held-out digits and their translated versions.

Weight sharing has a directly countable effect. The two convolutional layers together have 18,816 parameters, while the CNN’s first dense layer, Linear(9216, 128), has 1,179,776. Most parameters in this CNN are therefore in its dense head; shared filters do not by themselves ensure a small total model.

22.5 Summary

Moving the stroke moved its filter response: convolution is translation equivariant away from boundaries. Taking a global maximum erased its position, whereas local pooling retained some spatial information and offered no guarantee of an unchanged prediction. Stacked filters combine channels and enlarge the receptive field, so early layers detect simple features (edges, textures) and deeper layers combine them into complex patterns (shapes, objects). These are the architectural ideas illustrated by the MNIST network. Attention, developed in Chapter 23, offers another way to combine information across positions.

Exercises

Pen-and-Pencil

Exercise 22.1 (Convolution Operation) For a 5×5 input image and a 3×3 kernel, compute the convolution.

Input:

1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25

Kernel:

1 0 -1
1 0 -1
1 0 -1

What is the output size with no padding and stride 1?
Calculate the top-left output value.
What type of feature does this kernel detect?
How would you modify the convolution to preserve the input size?

Exercise 22.2 (Pooling Layers) For a 4×4 feature map:

Apply 2×2 max pooling with stride 2. What is the output?
Apply 2×2 average pooling with stride 2. What is the output?
Why is pooling useful in CNNs?
What is global average pooling and when is it used?

Exercise 22.3 (CNN Architecture) Design a CNN for classifying 32×32 color images into 10 classes.

If the first conv layer has 32 filters of size 3×3, how many parameters does it have?
After a 3×3 conv (no padding) followed by 2×2 max pool (stride 2), what is the output size?
Write the architecture for a simple CNN with 2 conv layers and 1 FC layer.
Calculate the total number of parameters in your architecture.

Exercise 22.4 (True/False: CNNs)

A 5×5 convolution can be replaced by two 3×3 convolutions with fewer parameters.
Max pooling is differentiable everywhere.
In a CNN, the number of parameters in a conv layer depends on the input image size.
Data augmentation (flipping, rotating images) helps CNNs learn more robust features.
Modern CNNs use fully connected layers after every conv layer.

Exercise 22.5 (Receptive Field Growth: Stacked 3×3 vs. One Large Kernel) Recall from Chapter 22 that the receptive field is the region of the input that influences a single output value. Consider a stack of \(L\) convolutional layers, each with a \(3\times 3\) kernel, stride 1, and enough padding to keep the spatial size fixed. (Padding controls the output size, not the receptive field.)

Argue that each additional \(3\times 3\) layer widens the receptive field of a fixed output unit by one pixel on every side, and use this to derive a general formula for the receptive field \(\text{RF}(L)\) after \(L\) stacked layers. Check your formula for \(L = 1, 2, 3\).
Show that \(L=2\) stacked \(3\times 3\) layers has the same receptive field as a single \(5\times 5\) layer. Assuming \(C\) input and output channels throughout, compare the parameter counts \(2\times(3\times 3\times C^2)\) and \(5\times 5\times C^2\) (ignoring biases). Which is smaller, and by roughly what percentage?
Generalize part (b): show that \(L=3\) stacked \(3\times 3\) layers matches the receptive field of a single \(7\times 7\) layer, and compare the parameter counts \(3\times(3\times 3\times C^2)\) and \(7\times 7\times C^2\).
Beyond the parameter count, the stacked design inserts an extra nonlinearity (e.g., ReLU) after every layer instead of just one. Why does this matter, and why do modern CNN architectures generally favor stacks of small kernels over single large ones?

Computing

Exercise 22.6 (2D Convolution from Scratch) Implement 2D convolution (valid, no padding, stride 1) using only numpy, then use it to find a vertical edge in a small synthetic image. Here image is the \(6\times 6\) array whose first three columns are 0 and last three columns are 1, and sobel_x is the \(3\times 3\) Sobel kernel with rows \((-1, 0, 1)\), \((-2, 0, 2)\) and \((-1, 0, 1)\).

Write a function conv2d(image, kernel) that slides the kernel over the image with stride 1 and no padding, at each position summing the elementwise product of kernel and patch (as the conv() helper does in Chapter 22). For an \(H \times W\) image and a \(k \times k\) kernel with stride \(s\), give the general formula for the output height and width, and check that it matches the shape returned by your function on image and sobel_x.
Apply conv2d to image using sobel_x. Where is the response largest, and why is this the behavior you would expect from a vertical edge detector? Verify your output against scipy.signal.correlate2d(image, sobel_x, mode="valid"). Then compare with scipy.signal.convolve2d(image, sobel_x, mode="valid"), which rotates the kernel 180 degrees before sliding it, the textbook definition of convolution. Do the two scipy results agree here, and what does that say about sobel_x?
The kernel above has 9 weights plus 1 bias, reused at all 16 output positions. How many parameters would a fully connected layer need to map all 36 input pixels to the same 16 outputs? What does this comparison illustrate about parameter sharing in convolutional layers?

Exercise 22.7 (Max and Average Pooling from Scratch) Chapter 22 says that local max pooling can reduce sensitivity to some shifts but does not guarantee an unchanged prediction. This exercise implements \(2\times 2\) pooling with a chosen stride in numpy and checks exactly how much invariance that is. The input x is the \(2\times 6\) array whose two rows are both \((0, 1, 9, 0, 2, 0)\).

Write a function pool2d(x, reducer, size=2, stride=2) that slides a size-by-size window over x with the given stride (no padding) and fills each output entry with reducer applied to that window. Use it to define max_pool2d and avg_pool2d (with reducer equal to np.max and np.mean), apply both to x, and check the results against expected_max = [[1, 9, 2]] and expected_avg = [[0.5, 4.5, 1.0]] with np.allclose.
Build x_shift by shifting x one pixel to the right (fill the new first column with zeros, and drop the last column). Pool x_shift with both functions and compare, entry by entry, to the part (a) results.
The two large activations in x, the values 9 and 2, sit at columns 2 and 4. Using the part (b) comparison, explain why max pooling reproduces both of them exactly after the shift while average pooling does not, and relate this to why max pooling, rather than average pooling, is usually credited with giving CNNs a degree of translation invariance.

Exercise 22.8 (Training and Shrinking a CNN on MNIST in PyTorch) Chapter 22 trains a PyTorch convolutional network on 8,000 MNIST images and reports 93% test accuracy, against 77% for the fully connected network. This exercise reproduces that pipeline, then shrinks the convolutional channels and measures what the shrinking costs. Use the same data and split as the chapter: load the MNIST test files with the chapter’s parse_images and parse_labels, then reuse numpy.random.RandomState(0) to build traini (the first 8,000 shuffled indices) and testi (the remaining 2,000). The chapter’s training code calls torch.manual_seed(0) before building the model; do the same before building each model here, so that your numbers are reproducible.

Implement the chapter’s Net: conv1 mapping 1 to 32 channels and conv2 mapping 32 to 64 channels (both 3×3 valid convolutions, each followed by ReLU), then a single 2×2 max pool, a flatten to 9216 features, fc1 mapping 9216 to 128 (with ReLU), and fc2 mapping 128 to 10 with a log-softmax output. Train for 5 epochs at batch size 128, minimizing F.nll_loss with torch.optim.Adadelta at lr=0.1. Print the training loss at the end of each epoch and report the final test accuracy on testi.
Build a smaller network identical to Net except that conv1 outputs 16 channels and conv2 outputs 32, so the flattened size becomes 32×12×12 = 4608 and fc1 maps 4608 to 128. Train it with the same data split, batch size, epoch count, and optimizer settings. Report its loss trajectory and final test accuracy.
For both networks, compute the total parameter count with sum(p.numel() for p in model.parameters()) and the count for fc1 alone. What fraction of each network’s parameters sits in fc1? By roughly what factor did halving the channel counts reduce the total?
Compare the two networks’ accuracies and parameter counts, to each other and to the chapter’s 77% fully connected baseline. Does halving the channels cost much accuracy? Using your answer to (c), explain why shrinking the convolutional channels cuts the total parameter count by roughly the same factor even though the convolutional layers themselves hold under 2% of the parameters, and connect this to Exercise 22.5’s point about where a CNN’s real parameter cost lives.