Math for CV Beginners: Vectors, Matrices, Convolutions

You do not need a math degree to do computer vision. You need about five ideas, and you need them cold, because they show up in every single method after this. This post works through all five with numbers you can check by hand.

Prerequisites: What is computer vision. If you want a refresher on vectors and matrices in general, scalars and vectors and matrix operations cover the wider ground.

Idea 1: shapes

Everything starts with knowing what shape your data is. A grayscale image is a 2D array. A colour image is 3D. A batch of colour images is 4D.

Two things trip everyone up.

Rows come before columns. img.shape gives (height, width), but almost every drawing function takes (x, y) which is (column, row). So img.shape[0] is height and cv2.circle wants the width coordinate first. Mixing these up produces images that look transposed or drawings that land in the wrong place.

OpenCV puts channels last, PyTorch puts them first. OpenCV uses HWC (height, width, channel). PyTorch uses CHW. Converting between them is a transpose:

import numpy as np

img_hwc = np.zeros((480, 640, 3), dtype=np.uint8)   # OpenCV layout
img_chw = np.transpose(img_hwc, (2, 0, 1))          # PyTorch layout
print(img_hwc.shape, img_chw.shape)   # (480, 640, 3) (3, 480, 640)

And one more: OpenCV reads colour as BGR, not RGB. If your image looks blue where it should be orange, this is why.

import cv2
bgr = cv2.imread("photo.jpg")            # B, G, R order
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)

Idea 2: convolution, by hand

convolution Slide a small grid of weights (a kernel) over an image. At each position, multiply overlapping values pairwise and add them up. The sum becomes one output pixel. is the single most important operation in computer vision. It powers blurring, sharpening, edge detection, and every convolutional neural network.

Let’s do one completely by hand.

The input

A 5×5 grayscale patch with a vertical edge: dark on the left, bright on the right.

Position by position

The kernel is 3×3, so it fits in the 5×5 input at 3 horizontal positions and 3 vertical positions. That is why the output is 3×3.

Output position (0, 0). The kernel covers input rows 0–2, columns 0–2:

Multiply each value by the matching kernel weight and add:

$(-1)(20) + (0)(20) + (1)(20) = -20 + 0 + 20 = 0$ $(-2)(20) + (0)(20) + (2)(20) = -40 + 0 + 40 = 0$ $(-1)(20) + (0)(20) + (1)(20) = -20 + 0 + 20 = 0$

Total: $0 + 0 + 0 = \mathbf{0}$. Flat region, no edge, zero response. Good.

Output position (0, 1). Shift one column right. The kernel now covers columns 1–3:

$(-1)(20) + (0)(20) + (1)(220) = -20 + 220 = 200$ $(-2)(20) + (0)(20) + (2)(220) = -40 + 440 = 400$ $(-1)(20) + (0)(20) + (1)(220) = -20 + 220 = 200$

Total: $200 + 400 + 200 = \mathbf{800}$. Strong positive response. The kernel found the edge.

Output position (0, 2). Columns 2–4:

$(-1)(20) + (0)(220) + (1)(220) = 200$ $(-2)(20) + (0)(220) + (2)(220) = 400$ $(-1)(20) + (0)(220) + (1)(220) = 200$

Total: $\mathbf{800}$ again.

Every row of the input is identical, so every row of the output is identical. Final output:

$\begin{bmatrix} 0 & 800 & 800 \\ 0 & 800 & 800 \\ 0 & 800 & 800 \end{bmatrix}$

Reading the result

Three things worth noticing, because they explain behaviour you will see constantly.

The edge is one column wide, but the response is two columns wide. A 3×3 kernel “feels” the edge from one column before it and one column after. This blur is why raw Sobel output looks thick and why Canny edge detection adds a thinning step.

The response is 800, far outside the 0–255 range. Convolution output is not an image. It is a signed number that can be negative or huge. If you save it directly as uint8 it wraps around and you get garbage. Always keep filter output in a float or signed type until you deliberately rescale it.

Flip the input (bright on the left) and the response becomes −800. The sign tells you the direction of the edge, not just that one exists. Take the absolute value if you only care about edge strength.

The response profile across the row:

The blue line is the brightness. The red line is what the filter computed. Where brightness is flat, the response is zero. Where brightness jumps, the response spikes. That is all an edge detector does.

Slide the kernel yourself

Step through the positions and watch the window move. The count of stopping positions is exactly the output size, which is the next idea.

Try different kernels on the same input

Pick “Blur (Average)” and the output looks like a softened copy. Pick “Edge (Vertical)” and only vertical boundaries light up. Same input, same sliding operation, completely different result. The only thing that changed is nine numbers in the kernel.

That is the key insight for neural networks: the kernel numbers are the only thing that matters, so let the model learn them instead of you choosing them.

Idea 3: the output size formula

Before running anything, you can compute exactly how big the output will be:

$W_{out} = \left\lfloor \frac{W_{in} - F + 2P}{S} \right\rfloor + 1$

where $W_{in}$ is input width, $F$ is kernel size, $P$ is padding added to each side, and $S$ is the stride (how far the kernel jumps each step).

Three worked cases

Case 1: our example above. $W_{in} = 5$, $F = 3$, $P = 0$, $S = 1$:

$W_{out} = \left\lfloor \frac{5 - 3 + 0}{1} \right\rfloor + 1 = 2 + 1 = 3 \;\checkmark$

Matches the 3×3 output we computed by hand.

Case 2: “same” padding. $W_{in} = 224$, $F = 3$, $P = 1$, $S = 1$:

$W_{out} = \left\lfloor \frac{224 - 3 + 2}{1} \right\rfloor + 1 = 223 + 1 = 224$

Input size equals output size. This is why nearly every modern CNN uses 3×3 kernels with padding 1: the spatial size only changes where you deliberately want it to.

Case 3: the ResNet stem. $W_{in} = 224$, $F = 7$, $P = 3$, $S = 2$:

$W_{out} = \left\lfloor \frac{224 - 7 + 6}{2} \right\rfloor + 1 = \left\lfloor \frac{223}{2} \right\rfloor + 1 = 111 + 1 = 112$

A stride of 2 halves the resolution. That is the standard way networks shrink a feature map, and you will see it again in CNN architectures.

To keep the size unchanged with an odd kernel, set $P = (F-1)/2$. So $F=3$ needs $P=1$, $F=5$ needs $P=2$, $F=7$ needs $P=3$. That pattern covers almost every layer you will write.

Idea 4: normalization

Raw pixels run from 0 to 255. That range causes two concrete problems.

Problem 1: overflow. uint8 arithmetic wraps around.

import numpy as np
a = np.array([200], dtype=np.uint8)
print(a + 100)     # [44]  — 300 wrapped around to 44
print(a.astype(np.float32) + 100)   # [300.]  correct

Problem 2: scale. Gradient-based training works badly when inputs are large and all positive. Every weight gets pushed in the same direction on every step.

The fix is two steps: scale to 0–1, then standardize per channel.

$x_{scaled} = \frac{x}{255}, \qquad x_{norm} = \frac{x_{scaled} - \mu}{\sigma}$

Worked example

Take a pixel with value 200 in the red channel. Using the standard ImageNet statistics ($\mu_R = 0.485$, $\sigma_R = 0.229$):

$x_{scaled} = \frac{200}{255} = 0.784$ $x_{norm} = \frac{0.784 - 0.485}{0.229} = \frac{0.299}{0.229} = 1.306$

The pixel is 1.31 standard deviations brighter than the average red value in ImageNet.

Now a dark pixel, value 30:

$x_{scaled} = \frac{30}{255} = 0.118, \qquad x_{norm} = \frac{0.118 - 0.485}{0.229} = -1.603$

After normalization the values are centred near zero and mostly land between −2 and +2, which is the range optimizers are happiest with.

mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std  = np.array([0.229, 0.224, 0.225], dtype=np.float32)

def normalize(rgb_uint8):
    x = rgb_uint8.astype(np.float32) / 255.0
    return (x - mean) / std

Use the ImageNet numbers when you fine-tune a pretrained model, because that is what its weights expect. Compute your own dataset’s mean and standard deviation when you train from scratch, especially for medical or satellite images whose brightness looks nothing like ImageNet photos.

Idea 5: 2D transforms as matrices

Rotating, scaling, and moving an image are all matrix multiplications once you write points the right way.

A point $(x, y)$ becomes a 3-element homogeneous coordinate A 2D point written as (x, y, 1) so that translation, which is an addition, can be expressed as part of a matrix multiplication. vector $(x, y, 1)$. Then any affine transform is a single 3×3 matrix:

$\begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} a & b & t_x \\ c & d & t_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}$

The top-left 2×2 block handles rotation, scaling, and shear. The right column handles the shift.

Worked example: rotate 90° then shift

Rotating counter-clockwise by angle $\theta$ uses:

$R = \begin{bmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{bmatrix}$

At $\theta = 90°$, $\cos 90° = 0$ and $\sin 90° = 1$, so:

$R = \begin{bmatrix} 0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}$

Apply it to the point $(100, 50)$:

$\begin{bmatrix} 0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} 100 \\ 50 \\ 1 \end{bmatrix} = \begin{bmatrix} (0)(100) + (-1)(50) + 0 \\ (1)(100) + (0)(50) + 0 \\ 1 \end{bmatrix} = \begin{bmatrix} -50 \\ 100 \\ 1 \end{bmatrix}$

The point moved to $(-50, 100)$. It is off the image now, which is exactly what happens when you rotate about the origin instead of the image centre. Add a translation of $t_x = 200$ to bring it back:

$\begin{bmatrix} 0 & -1 & 200 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} 100 \\ 50 \\ 1 \end{bmatrix} = \begin{bmatrix} -50 + 200 \\ 100 \\ 1 \end{bmatrix} = \begin{bmatrix} 150 \\ 100 \\ 1 \end{bmatrix}$

In OpenCV you rarely build these by hand:

import cv2

h, w = img.shape[:2]
centre = (w / 2, h / 2)
M = cv2.getRotationMatrix2D(centre, angle=30, scale=1.0)   # returns 2x3
rotated = cv2.warpAffine(img, M, (w, h))

getRotationMatrix2D builds the rotation and the translation needed to keep the centre fixed, which is what you almost always want.

Why this matters beyond rotation: the same idea with a full 3×3 matrix (no fixed bottom row) is a homography A 3x3 matrix that maps one flat plane in an image to another. It handles perspective, not just rotation and scaling. , which is how you flatten a photographed document, stitch a panorama, or correct a camera looking at a plane at an angle. That is the whole subject of camera calibration and perspective correction.

Bonus: mean vs median, and why filters differ

One more piece of arithmetic that explains a lot of behaviour later. Take a 3×3 window with one bad pixel from sensor noise:

Mean filter. Sum is $50 + 52 + 240 + 48 + 50 + 51 + 51 + 49 + 52 = 643$. Divide by 9:

$\text{mean} = \frac{643}{9} = 71.4$

Median filter. Sort the nine values: 48, 49, 50, 50, 51, 51, 52, 52, 240. The middle one is:

$\text{median} = 51$

The true value in that region is around 50. The mean was dragged to 71 by a single outlier. The median ignored it completely.

Median filtering is not a convolution, because sorting cannot be written as multiply-and-add. That distinction matters: convolutions can be stacked and optimized on a GPU, and non-linear filters cannot. You will use both in image fundamentals.

Where each idea shows up later

graph TD
  S["Shapes and channel order"] --> A["Every post. Bugs here waste days."]
  C["Convolution"] --> B["Edge detection, blurring, CNNs, feature maps"]
  O["Output size formula"] --> D["Designing and debugging CNN layers"]
  N["Normalization"] --> E["Training stability, transfer learning, deployment parity"]
  T["Matrix transforms"] --> F["Warping, calibration, stitching, augmentation, 3D vision"]

Practice task

Work these out on paper, then check them in code. The point is to trust the arithmetic before you trust a library.

1. Apply the Sobel-Y kernel (the transpose of Sobel-X) to the same 5×5 input from earlier. Predict the output before computing it. Why is it all zeros?
2. A layer takes a $(1, 3, 256, 256)$ input, uses a 5×5 kernel with padding 2, stride 2, and 32 output channels. What is the output shape? Verify with the formula.
3. Compute the normalized value of a blue-channel pixel with raw value 90, using $\mu_B = 0.406$ and $\sigma_B = 0.225$.
4. Build the matrix that flips an image vertically when the image height is 480, then apply it to the point $(200, 100)$.

import torch, torch.nn as nn
x = torch.zeros(1, 3, 256, 256)
layer = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=5, padding=2, stride=2)
print(layer(x).shape)   # check your answer to question 2

Answers: (1) all zeros, because the input has no horizontal edge — every column is constant top to bottom. (2) $(1, 32, 128, 128)$. (3) $-0.242$. (4) $(200, 380)$.

Summary

Five ideas cover the math you need for the rest of this series.

Shapes tell you what your data looks like, and channel order (HWC vs CHW, BGR vs RGB) is the top source of silent bugs. Convolution is multiply-and-add over a sliding window, and you worked one out cell by cell. The output size formula lets you design layers on paper. Normalization moves pixels from 0–255 into a range that optimizers can work with, and the same numbers must be used at training and at inference. Affine matrices with homogeneous coordinates turn rotate, scale, and shift into a single multiplication, which generalizes to the homographies used in calibration and stitching.

What comes next

Next up is image fundamentals, which puts this arithmetic to work on real images: colour spaces, histograms that reveal exposure problems, and the filters that clean noise without destroying the edges you need. Then OpenCV setup gets you a working environment and ten hands-on tasks.