Optical Flow and Motion Tracking in Video

Stereo matched two cameras at one instant. Optical flow matches one camera across two instants. It is the same problem with the axis rotated, and it gives you motion: which way things moved, how fast, and whether anything moved at all.

Prerequisites: Stereo vision for the matching idea, and image gradients.

The problem: measuring traffic at a junction

A council wants vehicle counts and average speeds at a junction. A camera is already mounted on the pole. Detecting cars is easy enough, but a detector run frame by frame gives no identity — is the car in frame 91 the same car as in frame 90, or a new one?

Motion answers it. If you know how each part of the image moved between frames, you can follow a car across the whole scene, count it once, and measure how fast it travelled.

graph LR
  A["Frame t"] --> C["Estimate motion<br/>of each tracked point"]
  B["Frame t+1"] --> C
  C --> D["Link points into tracks"]
  D --> E["Count line crossings"]
  D --> F["Pixels/frame → m/s<br/>using fps and ground scale"]

Brightness constancy

The whole field rests on one assumption: a point keeps its brightness as it moves.

$I(x, y, t) = I(x + \Delta x,\; y + \Delta y,\; t + \Delta t)$

Expand the right side as a first-order Taylor series and cancel:

$I_x u + I_y v + I_t = 0$

The equation is intuitive if you read it aloud. If a pixel got darker over time ($I_t$ negative) and brightness rises to the right ($I_x$ positive), then whatever was here must have moved right, bringing the darker stuff in.

The aperture problem

One equation, two unknowns. That is not a solvable system, and it is not a technicality — it is a real, visible limitation.

Suppose at some pixel $I_x = 10$, $I_y = 0$, $I_t = -30$:

$10u + 0v - 30 = 0 \;\Rightarrow\; u = 3, \quad v = \text{anything}$

The horizontal motion is pinned at 3 pixels. The vertical motion is completely undetermined. Moving 3 right, or 3 right and 50 down, or 3 right and 50 up all produce exactly the same brightness change.

Look at those two frames. The edge moved one pixel right — but did the object also move up or down? Through this small window you cannot tell, because the edge looks identical everywhere along its length.

This is exactly the corner-versus-edge distinction from feature matching, arriving again from a different direction. Only corners give unambiguous motion.

Lucas-Kanade: use a window

Kanade and Lucas made one assumption that fixes everything: all pixels in a small window move together. A 5×5 window gives 25 copies of the flow equation sharing the same $u$ and $v$ — 25 equations for 2 unknowns, comfortably over-determined.

Solving by least squares gives the normal equations:

$\begin{bmatrix} \sum I_x^2 & \sum I_x I_y \\ \sum I_x I_y & \sum I_y^2 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} = -\begin{bmatrix} \sum I_x I_t \\ \sum I_y I_t \end{bmatrix}$

Work one through. For a 5×5 window on a car’s number plate corner, the sums come out as $\sum I_x^2 = 400$, $\sum I_x I_y = 120$, $\sum I_y^2 = 300$, $\sum I_x I_t = -900$, $\sum I_y I_t = -600$:

$\begin{bmatrix} 400 & 120 \\ 120 & 300 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} 900 \\ 600 \end{bmatrix}$

The determinant is:

$\det = (400)(300) - (120)(120) = 120{,}000 - 14{,}400 = 105{,}600$

Solve with Cramer’s rule:

$u = \frac{(900)(300) - (120)(600)}{105{,}600} = \frac{270{,}000 - 72{,}000}{105{,}600} = \frac{198{,}000}{105{,}600} = 1.875$

$v = \frac{(400)(600) - (120)(900)}{105{,}600} = \frac{240{,}000 - 108{,}000}{105{,}600} = \frac{132{,}000}{105{,}600} = 1.25$

Check by substituting back:

$400(1.875) + 120(1.25) = 750 + 150 = 900 \;\checkmark$ $120(1.875) + 300(1.25) = 225 + 375 = 600 \;\checkmark$

This point moved 1.875 pixels right and 1.25 pixels down between frames.

Knowing in advance whether a point is trackable

That 2×2 matrix on the left is the structure tensor The 2x2 matrix of summed squared image gradients over a window. Its two eigenvalues describe how much the patch varies in each of two perpendicular directions. , and its eigenvalues tell you whether the solve is trustworthy — before you rely on the answer.

For our matrix, trace = 700 and det = 105,600:

$\lambda = \frac{700 \pm \sqrt{700^2 - 4(105{,}600)}}{2} = \frac{700 \pm \sqrt{490{,}000 - 422{,}400}}{2} = \frac{700 \pm 260}{2}$

$\lambda_1 = 480, \qquad \lambda_2 = 220$

Both large. This is a corner, and its motion is well determined in every direction.

Now compare three patch types:

This is precisely what cv2.goodFeaturesToTrack does: it computes the structure tensor everywhere and keeps the points where the smaller eigenvalue $\lambda_2$ is largest. That is the Shi-Tomasi criterion.

import cv2, numpy as np

prev_gray = cv2.cvtColor(prev, cv2.COLOR_BGR2GRAY)

p0 = cv2.goodFeaturesToTrack(prev_gray,
                             maxCorners=300,
                             qualityLevel=0.01,   # relative to the best λ2 found
                             minDistance=10,      # spread points out
                             blockSize=7)

Pyramids: handling fast motion

Lucas-Kanade uses a first-order Taylor expansion, which is only accurate for small displacements. A raw 5×5 window handles about 1–2 pixels of motion. A car at 50 km/h across a junction camera moves 30 pixels per frame.

The fix is to shrink the image. At half resolution, a 32-pixel motion becomes 16 pixels. Halve again: 8. Again: 4 — now within Lucas-Kanade’s range.

Solve at the coarsest level, scale the answer up by 2, use it as the starting guess at the next level, and refine. Each level only has to correct a small residual, which is exactly what Lucas-Kanade is good at.

lk_params = dict(
    winSize=(21, 21),
    maxLevel=3,                      # 4 levels total: 0, 1, 2, 3
    criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 30, 0.01),
)

p1, status, err = cv2.calcOpticalFlowPyrLK(prev_gray, gray, p0, None, **lk_params)

good_new = p1[status == 1]
good_old = p0[status == 1]

More levels are not free. At the coarsest level small objects have been blurred away entirely, so a fast-moving pedestrian’s leg may not exist there and the tracker follows the torso instead. Set maxLevel from your actual maximum expected motion rather than maximising it.

Sparse versus dense flow

Lucas-Kanade tracks specific points. Dense methods compute flow for every pixel.

# Dense flow: an (H, W, 2) array of (dx, dy) per pixel
flow = cv2.calcOpticalFlowFarneback(
    prev_gray, gray, None,
    pyr_scale=0.5, levels=3, winsize=15,
    iterations=3, poly_n=5, poly_sigma=1.2, flags=0)

mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])

# Standard visualisation: hue = direction, value = speed
hsv = np.zeros_like(prev)
hsv[..., 0] = ang * 180 / np.pi / 2
hsv[..., 1] = 255
hsv[..., 2] = cv2.normalize(mag, None, 0, 255, cv2.NORM_MINMAX)
vis = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)

For the junction, sparse is the right choice. You care about a few hundred vehicle corners, not every blade of grass on the verge, and 3 ms per frame leaves headroom for detection on the same CPU.

From pixels per frame to km/h

Flow gives pixels per frame. Two more numbers convert it to real speed.

$\text{speed} = \frac{\text{flow (px/frame)} \times \text{fps}}{\text{pixels per metre}}$

A car’s tracked corners average 12 px/frame. The camera runs at 30 fps. Painted lane markings, known to be 3 m long, span 60 pixels on the road surface, so the scale is 20 px/m:

$\text{speed} = \frac{12 \times 30}{20} = \frac{360}{20} = 18 \text{ m/s} = 64.8 \text{ km/h}$

That last row matters in practice. A tracking failure — a point jumping from one car to another — produces a large fake flow that looks exactly like a speeding vehicle. Bound your outputs by what is physically plausible and discard the rest.

The scale factor is the weak link. 20 px/m holds only at the ground plane and only in the part of the image where you measured it. A car further up the road has fewer pixels per metre because of perspective, so the same real speed reads as a lower flow. Either restrict measurement to a small calibrated patch of road, or rectify the road surface to a top-down view with a homography first — the technique from camera calibration.

Building a tracker

Points get lost. Objects go behind poles, leave the frame, or drift off their feature. A tracker needs upkeep.

def update_tracks(prev_gray, gray, p0, min_points=80):
    p1, status, err = cv2.calcOpticalFlowPyrLK(prev_gray, gray, p0, None, **lk_params)

    # forward-backward check: track back and see if we return to the start
    p0r, _, _ = cv2.calcOpticalFlowPyrLK(gray, prev_gray, p1, None, **lk_params)
    fb_error = abs(p0 - p0r).reshape(-1, 2).max(axis=1)

    keep = (status.ravel() == 1) & (fb_error < 1.0)
    p1 = p1[keep]

    # top up when too many are lost
    if len(p1) < min_points:
        extra = cv2.goodFeaturesToTrack(gray, maxCorners=300 - len(p1),
                                        qualityLevel=0.01, minDistance=10)
        if extra is not None:
            p1 = np.vstack([p1, extra])
    return p1

The forward-backward check is the single most valuable line here. Track a point forward to the next frame, then track it back. A good point returns to within a pixel of where it started. A point that latched onto the wrong thing does not, and this catches it with no extra information needed.

Row 2 deserves attention. Brightness constancy is an assumption, and outdoor video breaks it constantly — clouds, headlights, auto-exposure adjusting between frames. When every track fails at once, suspect illumination rather than the tracker.

Practice task

Record 30 seconds of traffic from a footbridge or a window overlooking a road.

1. Run goodFeaturesToTrack on the first frame and draw the points. Where do they land, and where do they not?
2. Track with calcOpticalFlowPyrLK at maxLevel=0. Count how many survive 30 frames.
3. Repeat at maxLevel=3. Compare the survival counts.
4. Add the forward-backward check with a threshold of 1.0 px. How many more get rejected, and do they look like real failures?
5. Measure a known length on the road, such as a lane marking, and compute pixels per metre.
6. Convert flow to km/h for ten vehicles. Compare with your own estimate by eye.
7. Now measure a vehicle near the camera and one far up the road that are clearly travelling at the same speed. Compare the numbers.

Step 7 is the point of the exercise. The far vehicle will read substantially slower, and the size of that gap is your perspective error. Once you have seen it, the case for rectifying the road plane first stops being theoretical.

Summary

Optical flow rests on brightness constancy, which linearises to $I_x u + I_y v + I_t = 0$: one equation, two unknowns. With $I_x = 10$, $I_y = 0$, $I_t = -30$ you get $u = 3$ and $v$ entirely undetermined — the aperture problem, and the reason only corners give reliable motion.

Lucas-Kanade assumes a window moves as one, producing 25 equations from a 5×5 patch. You solved a real system and got $u = 1.875$, $v = 1.25$, then checked both rows. The structure tensor’s eigenvalues, 480 and 220 in that case, tell you in advance whether the answer is trustworthy: both large is a corner, one large is an edge, both small is flat.

Pyramids extend the range from about 2 pixels to 32 by solving coarse and refining fine. Converting to real units needs frame rate and ground scale: 12 px/frame at 30 fps and 20 px/m is 18 m/s, or 64.8 km/h — but only if the scale holds where you measured, which perspective makes false across a full frame.

What comes next

Two directions from here.

If you want to keep working with motion classically, the flow field you just computed is the raw material for motion descriptors. Given (u, v) at each pixel, the magnitude is sqrt(u² + v²) and the direction is atan2(v, u), and histogramming those over a grid of cells gives HOF — a description of what is moving and where, which a classifier can consume directly. Taking spatial gradients of the flow field instead gives MBH, which cancels camera motion entirely. Both are covered in HOG, HOF and MBH descriptors.

If you want the modern treatment, video understanding covers learned flow networks, frame interpolation, stabilisation, temporal denoising, and how to run detection on video without paying full inference cost on every frame.

That completes the classical half of the series. You can now find edges, measure shapes, match points between images, calibrate a camera, recover depth, and measure motion — all without a single trained model, and with every step explainable.

The next track changes approach. Your first image classifier with PyTorch and transfer learning starts from a different premise: instead of designing the features yourself, you let the model learn them from labelled examples. The classical methods do not become obsolete. They remain the fastest, cheapest, and most explainable option whenever the problem is geometric, and they are often the right preprocessing step before a network ever sees the image.