22 Convolutional Neural Networks

“The face is not a simple pattern; it is a hierarchy of parts within parts within parts.”—David Marr, Vision (1982)

The feed-forward networks of Chapter 18 treat every input dimension identically. When the input is an image, this ignores spatial structure: nearby pixels are more related than distant ones, and the same pattern (an edge, a texture) can appear anywhere in the frame. Convolutional neural networks encode these two priors, locality and translation invariance, directly into the architecture.

The foundational architecture for convolutional neural networks emerged in 1980, when Kunihiko Fukushima introduced the Neocognitron (Fukushima 1980), a hierarchical neural network inspired by Hubel and Wiesel’s discoveries about simple and complex cells in the mammalian visual cortex. The Neocognitron was the first architecture to use convolutional layers for feature extraction and pooling layers for translation invariance, successfully recognizing handwritten Japanese characters through unsupervised learning. In 1988, Wei Zhang and colleagues applied backpropagation to train a simplified Neocognitron for alphabet recognition (Zhang et al. 1988), demonstrating that supervised learning could be effectively combined with the convolutional architecture. This combination of Fukushima’s architectural insights with gradient-based supervised learning became the foundation for modern CNNs, which now power everything from facial recognition on smartphones to autonomous vehicle perception systems.

This chapter introduces CNNs by first demonstrating the limitations of fully connected networks on image data, then building intuition for how convolutions extract local features. We will work with the MNIST dataset—the same handwritten digit recognition.

22.1 The MNIST Dataset

The MNIST dataset contains grayscale images of handwritten digits, each 28×28 pixels. It serves as the “Hello World” of computer vision, a standard benchmark that is simple enough to train quickly but complex enough to demonstrate the mechanics of image classification. The original dataset includes 60,000 training images and 10,000 test images. For simplicity, we will use only the test dataset in our examples. Each image is labeled with the digit it represents (0-9), making this a 10-class classification problem.

Image data are represented as a matrix of pixel values. A grayscale image of size 28×28 pixels is a matrix of shape (28, 28), where each entry contains an intensity value from 0 (black) to 255 (white). Color images add a third dimension for the red, green, and blue channels, resulting in shape (28, 28, 3). Figure 22.1 shows some sample MNIST digits.

22.2 Fully Connected Approach

A natural starting point for image classification is a fully connected neural network, which treats the image as a flat vector of independent pixels. The input to the model is a 28×28 matrix, which is reshaped into a vector of size 784. The output is a vector of size 10, representing the probability of each class. We split the 10,000 images using 80% for training and 20% for validation, shuffling before splitting to avoid bias.

We use jax, a library for numerical computing similar to numpy but designed for accelerators like GPUs and TPUs. Training a neural network requires two core computational capabilities: automatic differentiation to compute gradients, and efficient linear algebra, particularly matrix multiplication, to propagate signals through layers. jax provides both, along with a jit function for compiling operations to run efficiently on accelerators.

The model is trained by minimizing the cross-entropy loss, and we use accuracy to evaluate the model.

Finally, we implement the feed-forward network with \(\tanh\) activation. The last layer is a linear layer with the \(z_i - \ln \left(\sum_j e^{z_j}\right)\) function applied component-wise. Thus, our predict function simply returns the logarithm of the probability of each class. Typically we would use the softmax function to compute the probability of each class. Given a vector of logits \(z\), the softmax function is defined as follows: \[ \sigma(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K}e^{z_j}} \] However, computation of the exponential of large numbers can lead to numerical instability (overflow). Instead, we work with the logarithm of the softmax function, which is more numerically stable: \[ \ln\sigma(z)_i = z_i - \ln\sum_{j=1}^{K}e^{z_j} \] This is exactly what we need to compute the cross-entropy loss.

The final architecture is shown in Figure 22.2 and achieves an accuracy of 77% on the test set and 80% on the training set.

graph LR
    subgraph Input["Input Layer"]
        I[784 nodes<br/>28×28 flattened image]
    end
    
    subgraph Hidden1["Hidden Layer 1"]
        H1[1024 nodes<br/>tanh activation]
    end
    
    subgraph Hidden2["Hidden Layer 2"]
        H2[1024 nodes<br/>tanh activation]
    end
    
    subgraph Output["Output Layer"]
        O[10 nodes<br/>log-softmax]
    end
    
    I -->|W1: 784×1024<br/>b1: 1024| H1
    H1 -->|W2: 1024×1024<br/>b2: 1024| H2
    H2 -->|W3: 1024×10<br/>b3: 10| O
    
    O --> P[Predicted digit<br/>0-9]
    
    style Input fill:#e1f5ff
    style Hidden1 fill:#fff4e1
    style Hidden2 fill:#fff4e1
    style Output fill:#e8f5e9
    style P fill:#f3e5f5

Figure 22.2: Fully Connected MNIST

Figure 22.3 shows the accuracy of the fully connected network on the test and training sets from 0 to 10 epochs.

Figure 22.3: Accuracy of the fully connected network on the test and training sets.

We can see that even a simple fully connected network achieves reasonable accuracy, but it has fundamental limitations. It treats each pixel independently, ignoring the spatial structure of the image and the fact that neighboring pixels are related. A fully connected layer connecting a \(28 \times 28\) image to 1024 hidden units requires over 800,000 parameters, most of which are redundant.

22.3 Convolutions

Convolutional neural networks address these limitations by exploiting two key properties of images:

Local connectivity: Useful features (edges, textures, shapes) are local: they depend on small neighborhoods of pixels, not the entire image.
Translation invariance: A feature detector useful in one part of an image is likely useful in other parts.

Key Concepts

A filter (also called a kernel) is a small matrix of learned weights, typically 3×3 or 5×5, that extracts local features from an image by computing a weighted sum of pixel values within a sliding window. When the filter’s pattern aligns with a pattern in the image, the dot product produces a high activation; when they don’t match, the activation is low.

Kernel size: The dimensions of the filter (e.g., 3×3). Odd sizes are preferred so the filter has a well-defined center.
Receptive field: The region of the input image that influences a single output value.
Stride: The number of pixels by which the filter shifts at each step. A stride of 1 moves pixel-by-pixel; a stride of 2 skips every other position, reducing the output size.
Padding: Adding zeros around the border of the input to control the output size. “Same” padding preserves the input dimensions; “valid” padding uses no padding, shrinking the output.

While mathematically identical to kernel smoothing in statistics, convolutions in deep learning differ in one critical aspect: the kernel weights are learned from data rather than predefined.

Let us look at a one-dimensional example. Suppose we have a one-dimensional input signal \(x\) and a one-dimensional filter \(w\). The convolution of \(x\) and \(w\) is defined as follows: \[ (x \ast w)(t) = \sum_{i=0}^{h-1}x(t+i)w(i), \] where \(h\) is the size of the filter. The convolution operation is used to filter the input signal.

To illustrate this, consider a simple example where we apply a moving average filter to a noisy sinusoidal signal. We generate a sine wave over the interval \([0, 10]\) and add Gaussian noise to simulate realistic measurement errors. The filter uses a uniform window of size 10, where each weight equals \(1/10\), effectively computing the local average of the signal. When we convolve this filter with the noisy signal, each output point becomes the weighted average of its neighboring input points, which smooths out the random fluctuations while preserving the underlying periodic structure. Figure 22.4 shows the original noisy signal in light gray and the smoothed result in black, demonstrating how convolution effectively extracts the signal from noise by exploiting local spatial structure.

Figure 22.4: Smoothing a noisy signal using 1D convolution with a moving average filter. The light gray line shows the original noisy sinusoidal signal, while the black line displays the smoothed output after applying a uniform filter of size 10.

A Toy Example

To build intuition, consider a simple example with 5×5 images and 3×3 filters. We will classify images into three classes: cross, diagonal, and right-diagonal.

We will use two filters.

The first filter has 1s along its main diagonal and 0s elsewhere. When placed over a matching diagonal line in the image, the dot product is maximized. The second filter detects the opposite diagonal. This is why convolutions are effective feature detectors: they respond strongly when their pattern matches what’s in the image.

We see that both outputs have pixels with high values. However, when we apply the same two filters to the diagonal image, we get different results.

The result of applying the second filter to the diagonal image is a matrix close to zero: the filter pattern doesn’t match the image content. Applying the second filter to the diagonal image yields near-zero activations, illustrating that the filter remains dormant when its specific search pattern is absent. This observation motivates the pooling layer: by retaining only the maximum activation within each region, the network preserves the presence of a detected feature while discarding its exact location. This provides a degree of translation invariance. Finally, we concatenate the pooled outputs and apply a fully connected layer for classification.

## Cross: [0.26163495 0.26163495 0.47673005]
## Diagonal: [0.5937724  0.08035836 0.32586923]
## Right-Diagonal: [0.08035836 0.5937724  0.32586923]

The model correctly predicted all three classes. This toy example illustrates the core CNN pipeline: convolutions extract local features, pooling provides spatial invariance, and fully connected layers perform classification.

22.4 CNN for MNIST

Now we apply these concepts to MNIST using PyTorch. While we used JAX for the fully connected network to show the low-level mechanics of gradients and matrix operations, we switch to PyTorch here to demonstrate a high-level deep learning framework. PyTorch provides powerful abstractions like nn.Conv2d and nn.Module that make building and debugging complex architectures like CNNs more intuitive, and it effectively handles the boilerplate of autograd and parameter management.

The architecture follows the same pattern: convolutional layers extract hierarchical features, pooling reduces spatial dimensions, and fully connected layers produce class probabilities.

The CNN uses the Adadelta optimizer, an adaptive learning rate method that adjusts step sizes based on gradient history. Unlike vanilla SGD (which we used for the fully connected network), Adadelta typically requires less hyperparameter tuning.

The CNN architecture is shown in Figure 22.5.

flowchart TB
    subgraph ConvLayers["Convolutional Layers"]
        direction LR
        I["Input<br/>1 × 28 × 28<br/>Grayscale MNIST"]
        C1["Conv2d: 1→32<br/>kernel: 3×3<br/>32 × 26 × 26"]
        R1["ReLU"]
        C2["Conv2d: 32→64<br/>kernel: 3×3<br/>64 × 24 × 24"]
        R2["ReLU"]
        P2["MaxPool2d<br/>2×2<br/>64 × 12 × 12"]
        F["Flatten<br/>9216 features"]

        I --> C1 --> R1 --> C2 --> R2 --> P2 --> F
    end
    
    subgraph FCLayers["Fully Connected<br/>Classification"]
        direction LR
        FC1L["Linear<br/>9216 → 128"]
        R3["ReLU"]
        FC2L["Linear<br/>128 → 10"]
        LS["Log-Softmax"]
        O["Output<br/>10 classes<br/>Digits 0-9"]
        
        FC1L --> R3 --> FC2L --> LS --> O
    end
    
    ConvLayers --> FCLayers

    %% Styling
    style I fill:#e1f5ff,stroke:#01579b,stroke-width:2px,color:#000
    style C1 fill:#fff9c4,stroke:#fbc02d,stroke-width:2px,color:#000
    style C2 fill:#fff9c4,stroke:#fbc02d,stroke-width:2px,color:#000
    style FC1L fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#000
    style FC2L fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#000
    style R1 fill:#fce4ec,stroke:#c2185b,stroke-width:2px,color:#000
    style R2 fill:#fce4ec,stroke:#c2185b,stroke-width:2px,color:#000
    style R3 fill:#fce4ec,stroke:#c2185b,stroke-width:2px,color:#000
    style P2 fill:#e0f2f1,stroke:#00695c,stroke-width:2px,color:#000
    style F fill:#e0f2f1,stroke:#00695c,stroke-width:2px,color:#000
    style LS fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000
    style O fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000

    style ConvLayers fill:#fafafa,stroke:#ccc,stroke-width:2px,color:#000
    style FCLayers fill:#fafafa,stroke:#ccc,stroke-width:2px,color:#000

Figure 22.5: CNN Architecture

This model achieves an accuracy of 93% on the test set, compared to 77% for the fully connected network. The convolutional layers use shared weights, leading to far fewer convolutional parameters than a fully connected network, though the first dense layer (Linear(9216, 128)) still accounts for most of the total parameter count. The convolutional layers share weights across spatial positions rather than learning separate weights for each pixel location. This parameter efficiency, combined with the inductive bias toward local and translation-invariant features, explains why CNNs have become the dominant architecture for computer vision tasks.

22.5 Summary

Convolutional neural networks exploit the spatial structure of images through three key mechanisms:

Convolutional layers apply learned filters that detect local patterns, sharing weights across the entire image.
Pooling layers reduce spatial dimensions while providing invariance to small translations.
Hierarchical feature learning: early layers detect simple features (edges, textures); deeper layers combine these into complex patterns (shapes, objects).

The CNN architecture we built achieves significantly higher accuracy than the fully connected network while using far fewer parameters. This parameter efficiency, combined with the inductive bias toward local and translation-invariant features, explains why CNNs have become the dominant architecture for computer vision tasks.

Exercises

Pen-and-Pencil

Exercise 22.1 (Convolution Operation) For a 5×5 input image and a 3×3 kernel, compute the convolution.

Input:

1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25

Kernel:

1 0 -1
1 0 -1
1 0 -1

What is the output size with no padding and stride 1?
Calculate the top-left output value.
What type of feature does this kernel detect?
How would you modify the convolution to preserve the input size?

Exercise 22.2 (Pooling Layers) For a 4×4 feature map:

Apply 2×2 max pooling with stride 2. What is the output?
Apply 2×2 average pooling with stride 2. What is the output?
Why is pooling useful in CNNs?
What is global average pooling and when is it used?

Exercise 22.3 (CNN Architecture) Design a CNN for classifying 32×32 color images into 10 classes.

If the first conv layer has 32 filters of size 3×3, how many parameters does it have?
After a 3×3 conv (no padding) followed by 2×2 max pool (stride 2), what is the output size?
Write the architecture for a simple CNN with 2 conv layers and 1 FC layer.
Calculate the total number of parameters in your architecture.

Exercise 22.4 (True/False: CNNs)

A 5×5 convolution can be replaced by two 3×3 convolutions with fewer parameters.
Max pooling is differentiable everywhere.
In a CNN, the number of parameters in a conv layer depends on the input image size.
Data augmentation (flipping, rotating images) helps CNNs learn more robust features.
Modern CNNs use fully connected layers after every conv layer.

Exercise 22.5 (Receptive Field Growth: Stacked 3×3 vs. One Large Kernel) Recall from Chapter 22 that the receptive field is the region of the input that influences a single output value. Consider a stack of \(L\) convolutional layers, each with a \(3\times 3\) kernel, stride 1, and enough padding to keep the spatial size fixed. (Padding controls the output size, not the receptive field.)

Argue that each additional \(3\times 3\) layer widens the receptive field of a fixed output unit by one pixel on every side, and use this to derive a general formula for the receptive field \(\text{RF}(L)\) after \(L\) stacked layers. Check your formula for \(L = 1, 2, 3\).
Show that \(L=2\) stacked \(3\times 3\) layers has the same receptive field as a single \(5\times 5\) layer. Assuming \(C\) input and output channels throughout, compare the parameter counts \(2\times(3\times 3\times C^2)\) and \(5\times 5\times C^2\) (ignoring biases). Which is smaller, and by roughly what percentage?
Generalize part (b): show that \(L=3\) stacked \(3\times 3\) layers matches the receptive field of a single \(7\times 7\) layer, and compare the parameter counts \(3\times(3\times 3\times C^2)\) and \(7\times 7\times C^2\).
Beyond the parameter count, the stacked design inserts an extra nonlinearity (e.g., ReLU) after every layer instead of just one. Why does this matter, and why do modern CNN architectures generally favor stacks of small kernels over single large ones?

Computing

Exercise 22.6 (2D Convolution from Scratch) Implement 2D convolution (valid, no padding, stride 1) using only numpy, then use it to find a vertical edge in a small synthetic image.

Write a function conv2d(image, kernel) that slides the kernel over the image with stride 1 and no padding, at each position summing the elementwise product of kernel and patch (as the conv() helper does in Chapter 22). For an \(H \times W\) image and a \(k \times k\) kernel with stride \(s\), give the general formula for the output height and width, and check that it matches the shape returned by your function on image and sobel_x.
Apply conv2d to image using sobel_x. Where is the response largest, and why is this the behavior you would expect from a vertical edge detector? Verify your output against scipy.signal.correlate2d(image, sobel_x, mode="valid"). Then compare with scipy.signal.convolve2d(image, sobel_x, mode="valid"), which rotates the kernel 180 degrees before sliding it, the textbook definition of convolution. Do the two scipy results agree here, and what does that say about sobel_x?
The kernel above has 9 weights plus 1 bias, reused at all 16 output positions. How many parameters would a fully connected layer need to map all 36 input pixels to the same 16 outputs? What does this comparison illustrate about parameter sharing in convolutional layers?

Exercise 22.7 (Max and Average Pooling from Scratch) The chapter’s toy CNN in Chapter 22 uses max pooling because it “provides a degree of translation invariance.” This exercise implements \(2\times 2\) pooling with a chosen stride in numpy and checks exactly how much invariance that is.

Write a function pool2d(x, reducer, size=2, stride=2) that slides a size-by-size window over x with the given stride (no padding) and fills each output entry with reducer applied to that window. Use it to define max_pool2d and avg_pool2d (with reducer equal to np.max and np.mean), apply both to x, and check the results against expected_max = [[1, 9, 2]] and expected_avg = [[0.5, 4.5, 1.0]] with np.allclose.
Build x_shift by shifting x one pixel to the right (fill the new first column with zeros, and drop the last column). Pool x_shift with both functions and compare, entry by entry, to the part (a) results.
The two large activations in x, the values 9 and 2, sit at columns 2 and 4. Using the part (b) comparison, explain why max pooling reproduces both of them exactly after the shift while average pooling does not, and relate this to why max pooling, rather than average pooling, is usually credited with giving CNNs a degree of translation invariance.

Exercise 22.8 (Training and Shrinking a CNN on MNIST in PyTorch) Chapter 22 trains a PyTorch convolutional network on 8,000 MNIST images and reports 93% test accuracy, against 77% for the fully connected network. This exercise reproduces that pipeline, then shrinks the convolutional channels and measures what the shrinking costs. Use the same data and split as the chapter: load the MNIST test files with the chapter’s parse_images and parse_labels, then reuse numpy.random.RandomState(0) to build traini (the first 8,000 shuffled indices) and testi (the remaining 2,000). The chapter never fixes PyTorch’s own random seed, so its exact figures drift a little from run to run; here we set torch.manual_seed(0) before building each model so the numbers below are reproducible. As with the chapter’s own mnist-cnn chunks, the training code is shown but not executed on render (#| eval: false), since a few epochs of CNN training is too slow to rerun on every build; the figures quoted in the solution come from actually running it.

Implement the chapter’s Net: conv1 mapping 1 to 32 channels and conv2 mapping 32 to 64 channels (both 3×3 valid convolutions, each followed by ReLU), then a single 2×2 max pool, a flatten to 9216 features, fc1 mapping 9216 to 128 (with ReLU), and fc2 mapping 128 to 10 with a log-softmax output. Train for 5 epochs at batch size 128, minimizing F.nll_loss with torch.optim.Adadelta at lr=0.1. Print the training loss at the end of each epoch and report the final test accuracy on testi.
Build a smaller network identical to Net except that conv1 outputs 16 channels and conv2 outputs 32, so the flattened size becomes 32×12×12 = 4608 and fc1 maps 4608 to 128. Train it with the same data split, batch size, epoch count, and optimizer settings. Report its loss trajectory and final test accuracy.
For both networks, compute the total parameter count with sum(p.numel() for p in model.parameters()) and the count for fc1 alone. What fraction of each network’s parameters sits in fc1? By roughly what factor did halving the channel counts reduce the total?
Compare the two networks’ accuracies and parameter counts, to each other and to the chapter’s 77% fully connected baseline. Does halving the channels cost much accuracy? Using your answer to (c), explain why shrinking the convolutional channels cuts the total parameter count by roughly the same factor even though the convolutional layers themselves hold under 2% of the parameters, and connect this to Exercise 22.5’s point about where a CNN’s real parameter cost lives.