CNN Architectures Explained: From LeNet to ResNet

You have used ResNet-18 twice now without knowing what is inside it. This post opens it up — not as history, but as a sequence of specific problems and the specific fixes that solved them. Every claim here comes with the arithmetic.

Prerequisites: Your first image classifier and convolution basics.

Why convolution at all

A 224×224 colour image has 150,528 numbers. Connect that to even 1,000 hidden units with a fully connected layer:

$224 \times 224 \times 3 \times 1000 = 150{,}528{,}000 \text{ weights}$

150 million parameters for one layer, and it would still be a bad model: a cat shifted ten pixels right activates completely different weights, so the network has to learn “cat” separately at every position.

A convolution layer with 64 filters of size 3×3 on 3 input channels:

$3 \times 3 \times 3 \times 64 + 64 = 1{,}728 + 64 = 1{,}792 \text{ parameters}$

84,000 times fewer parameters, and the feature is detected wherever it appears. Those two properties — weight sharing and translation equivariance — are the entire argument for convolution.

Step the filter through the grid above. The same nine weights are reused at every position; only the patch underneath changes.

The parameter formula

$\text{params} = K \times K \times C_{in} \times C_{out} + C_{out}$

Nothing in that formula mentions image size. A 3×3 conv from 64 to 128 channels costs 73,856 parameters whether the input is 8×8 or 800×800.

The output size formula

$\text{out} = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1$

Receptive field: how much each unit sees

A single 3×3 conv sees a 3×3 patch. Stack another and each unit of the second layer sees a 3×3 region of the first layer, which itself spans 3×3 of the input — so it sees 5×5.

Each 3×3 layer adds 2 to the receptive field. With stride, growth is multiplicative:

$RF_{n} = RF_{n-1} + (K - 1) \prod_{i<n} S_i$

That final row explains something practical. By layer4, a unit’s receptive field exceeds the input, which is why fine-tuning only layer4 adapts whole-object reasoning while leaving edge and texture detectors alone.

Why 3×3 beat 5×5

VGG’s contribution was noticing that two 3×3 convs reach the same 5×5 receptive field as one 5×5 conv, and cost less. With $C$ channels in and out:

$\text{one } 5\times5: \; 5 \times 5 \times C \times C = 25C^2$ $\text{two } 3\times3: \; 2 \times (3 \times 3 \times C \times C) = 18C^2$

18 versus 25 is 28% fewer parameters, and there is a ReLU between the two 3×3 layers where the 5×5 has none — more nonlinearity for less cost.

With $C = 128$: one 5×5 costs 409,600 weights, two 3×3 cost 294,912.

Try the filter selector above. Edge, blur and Sobel are the kinds of low-level detectors a trained network’s first layer converges to on its own.

The architectures, and what each one fixed

The competition that drove all of this

Those architecture numbers came out of one benchmark, and seeing its full trajectory makes the pace of change concrete.

The ImageNet Large Scale Visual Recognition Challenge ran annually on 1.2 million training images across 1,000 classes. The headline metric was top-5 error: the fraction of images where the correct label was not among the model’s five best guesses.

Two things in that chart are worth sitting with. The 2011-to-2012 drop of 9.4 points is larger than the total improvement of the preceding two years, and it is the single event that redirected the field. And the crossing of the human line in 2015 is why the competition eventually stopped being interesting — once error is a third of a human’s, the remaining errors are mostly label ambiguity.

Top-1 accuracy tells the longer story, because it kept moving after top-5 saturated:

For the wider arc this benchmark sits inside — including what came before SIFT+FV and why the datasets themselves mattered as much as the models — see the history of computer vision.

The depth problem

By 2015 the obvious move was to stack more layers. It did not work.

Read the training error column carefully. It goes up. If this were overfitting, training error would fall while test error rose. Instead the deeper network cannot even fit the training data — an optimisation failure, not a generalisation one.

That is a mathematical absurdity on its face. A 56-layer network can represent everything a 20-layer network can: set the extra 36 layers to the identity. So the extra capacity is there and gradient descent simply cannot find it.

Why: gradients shrink multiplicatively

Backpropagation multiplies a gradient by each layer’s Jacobian on the way back. If each layer scales it by an average factor of 0.9:

$0.9^{50} = 0.00515$. The first layers barely move, so they stay near their random initialisation, and everything built on top of random features is handicapped.

The residual block

He and colleagues changed one thing. Instead of a block computing $H(x)$ directly, have it compute a correction to its input:

$y = F(x) + x$

The visualization above shows the two paths. Now differentiate:

$\frac{\partial y}{\partial x} = \frac{\partial F(x)}{\partial x} + 1$

That $+1$ is the entire fix. Even when $\partial F/\partial x$ shrinks toward zero, the gradient reaching $x$ is at least 1. Stack fifty of these and the gradient still arrives at the first layer intact, because along the skip path it is multiplied by 1 fifty times instead of by 0.9.

There is a second reading that is just as useful. If a block is not needed, $F(x) = 0$ makes it the identity, and driving weights toward zero is exactly what weight decay does anyway. The identity is now the easy default rather than something the network must learn.

graph TD
  X["x, 64 channels"] --> C1["3×3 conv 64→64"]
  C1 --> B1["BatchNorm"]
  B1 --> R1["ReLU"]
  R1 --> C2["3×3 conv 64→64"]
  C2 --> B2["BatchNorm"]
  B2 --> ADD(("+"))
  X -->|"identity skip"| ADD
  ADD --> R2["ReLU"]
  R2 --> Y["y, 64 channels"]

class BasicBlock(nn.Module):
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(cout)
        self.conv2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False)
        self.bn2   = nn.BatchNorm2d(cout)
        # projection only when shape changes
        self.skip = nn.Sequential()
        if stride != 1 or cin != cout:
            self.skip = nn.Sequential(
                nn.Conv2d(cin, cout, 1, stride, bias=False),
                nn.BatchNorm2d(cout))

    def forward(self, x):
        out = torch.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return torch.relu(out + self.skip(x))

Note bias=False on every conv. BatchNorm immediately subtracts the mean, which cancels any bias term, so it would be dead weight.

Bottleneck blocks

For deeper networks, ResNet uses a three-layer block that is far cheaper. The trick is 1×1 convolutions that squeeze the channel count before the expensive 3×3.

A plain block at 256 channels:

$2 \times (3 \times 3 \times 256 \times 256) = 2 \times 589{,}824 = 1{,}179{,}648$

The bottleneck version — reduce to 64, do the 3×3 there, expand back:

$\underbrace{1 \times 1 \times 256 \times 64}_{16{,}384} + \underbrace{3 \times 3 \times 64 \times 64}_{36{,}864} + \underbrace{1 \times 1 \times 64 \times 256}_{16{,}384} = 69{,}632$

$\frac{1{,}179{,}648}{69{,}632} = 16.9\times \text{ fewer parameters}$

Choosing a backbone

Compare EfficientNet-B0 and ResNet-50: nearly the same accuracy from a tenth of the FLOPs. But EfficientNet’s depthwise separable convolutions are memory-bandwidth heavy, so its real CPU latency is only about twice as fast, not ten times. FLOPs are a poor predictor of latency. Always measure on your target hardware.

Practice task

Using the classifier from post 17.

1. Print sum(p.numel() for p in model.parameters()) for ResNet-18, ResNet-50, and MobileNetV3-Small.
2. For ResNet-18’s layer1[0].conv1, verify the parameter count against $K^2 C_{in} C_{out}$ by hand.
3. Pass a 1×3×224×224 tensor through and print the shape after each stage. Check each against the output-size formula.
4. Time 100 forward passes on CPU for each backbone. Plot latency against parameter count.
5. Fine-tune all three on your dataset for the same epochs. Plot accuracy against latency.
6. Take one BasicBlock and delete the skip connection. Retrain and compare.
7. Build a 40-layer plain conv network with no skips and train it. Record the training accuracy.

Steps 6 and 7 are the ones that make it concrete. The 40-layer plain network will train worse than a 20-layer one on the same data, and watching that happen on your own machine turns the vanishing-gradient argument from a claim into something you have seen.

Summary

Convolution replaced 150 million fully connected weights with 1,792, because the same filter is reused at every position. Parameters are $K^2 C_{in} C_{out} + C_{out}$ and do not depend on image size at all.

Receptive field grows with depth — 3, then 5, then 7 for stacked 3×3 layers, reaching 435 pixels by ResNet-18’s layer4, which is larger than the input. Two 3×3 convs match a 5×5’s field for $18C^2$ instead of $25C^2$, a 28% saving plus an extra ReLU.

Depth then hit a wall. A 56-layer plain network had higher training error than a 20-layer one, which is an optimisation failure rather than overfitting: at $0.9^{50} = 0.005$, early layers received half a percent of the gradient. The residual block’s $y = F(x) + x$ makes $\partial y / \partial x = \partial F/\partial x + 1$, and that $+1$ keeps the gradient alive through any depth.

Bottleneck blocks squeeze 256 channels to 64 for the expensive 3×3, cutting a block from 1,179,648 to 69,632 parameters — 17× — which is how ResNet-50 is both deeper and smaller than VGG-16.

What comes next

Classification names one thing per image. Most real problems need to know where things are, and how many. The YOLO object detection pipeline builds on everything here — a backbone extracts features, then a detection head predicts boxes and classes at many positions at once, scored with the IoU and mAP metrics from earlier in the series.