Video Understanding: Flow Networks, Interpolation, Stabilisation

Run your image detector on every frame of a video and you have a working system that is worse than it needs to be. It is slow, because you are paying full inference cost 30 times a second on frames that are 98% identical. And it is unstable, because a box that appears in frames 1 and 3 but not in frame 2 produces a flicker no amount of confidence tuning fixes.

Video is not a stack of independent images. This post is about the things you can only do once you accept that.

Prerequisites: optical flow for the classical methods, and YOLO for per-frame detection.

The shared premise

Every technique here rests on one observation, so it is worth stating precisely.

At 30 frames per second, consecutive frames are separated by 33 milliseconds. In that time a person walking at 1.4 m/s moves about 4.6 cm. On a camera resolving 1 cm per pixel at that distance, that is roughly 5 pixels.

Everything else in the frame is unchanged.

Two consequences follow. Redundancy is an opportunity — you can reuse computation. And change is information — the difference between frames tells you things a single frame never could, including depth from parallax, object boundaries from differential motion, and of course what is actually happening.

Learned optical flow

Classical flow methods — Lucas–Kanade and Farnebäck — solve an optimisation per frame pair using hand-designed assumptions: brightness stays constant, motion is locally smooth. They break precisely where those assumptions do, which is on large displacements, at occlusions, and on textureless regions.

FlowNet in 2015 asked whether a network could just learn the mapping. Two architectures came out of it, and the contrast between them is genuinely instructive.

FlowNetSimple

Stack the two frames into a 6-channel input and run a standard encoder–decoder.

flowchart LR
  A["Frame t<br/>3 channels"] --> C["Concatenate<br/>6 channels"]
  B["Frame t+1<br/>3 channels"] --> C
  C --> D["Conv encoder<br/>downsampling"]
  D --> E["Deconv decoder<br/>upsampling"]
  E --> F["Flow field<br/>u, v per pixel"]
  style C fill:#eef

This works. It is also asking a lot. The network must learn, from data alone, that channels 0–2 and channels 3–5 are the same scene at different times and that the task is to match patches between them. Nothing in the architecture says so.

FlowNetCorr

The second design processes each frame separately, then introduces an explicit correlation layer.

flowchart TD
  A["Frame t"] --> C["Conv stack<br/>shared weights"]
  B["Frame t+1"] --> D["Conv stack<br/>shared weights"]
  C --> E["Features f₁"]
  D --> F["Features f₂"]
  E --> G["Correlation layer<br/>compare f₁(x) against<br/>f₂(x+d) for all d"]
  F --> G
  G --> H["Matching cost volume"]
  H --> I["Decoder"]
  I --> J["Flow field"]
  style G fill:#fee,stroke:#c00

For a feature vector at position x in frame 1, it computes a dot product against the feature at x + d in frame 2, for every displacement d within a search radius. If the radius is 20 pixels in each direction, that is 41 × 41 = 1681 similarity scores per position.

The decoder’s job then becomes reading off which displacement scored best, and cleaning up ambiguity — which is far easier than discovering the concept of matching from scratch.

The general lesson generalises well beyond flow: encoding known structure in the architecture beats hoping the network discovers it. Convolution encodes translation equivariance. Skip connections encode that a layer should be able to pass its input through. Correlation encodes that this is a matching problem.

Frame interpolation

Given frames at times t and t+1, synthesise the frame at t+0.5.

The naive approach is to average the two frames. This is always wrong, and it is instructive to see why on actual numbers. Here is one row across a bright object moving right by 4 pixels:

Averaging produces two half-strength copies of the object, one at each original position and nothing in between. That is the ghosting you see in cheap slow motion. Warping along the flow field moves the object to where it actually was at the intermediate time, producing one object at full intensity.

The real pipeline:

flowchart LR
  A["Frame t"] --> C["Estimate flow<br/>both directions"]
  B["Frame t+1"] --> C
  C --> D["Scale flow by 0.5"]
  D --> E["Warp both frames<br/>toward t+0.5"]
  E --> F["Blend, weighted by<br/>occlusion confidence"]
  F --> G["Refinement network<br/>fills disocclusions"]
  G --> H["Frame t+0.5"]
  style G fill:#efe

The refinement stage exists for one specific reason. When an object moves, it uncovers background that was hidden in frame t and, if it moved far enough, may be hidden again in t+1. Those pixels have no source in either frame. Warping cannot produce them because there is nothing to warp. Only a generative step can fill them, and it is guessing.

Video stabilisation

Shaky footage has two motions mixed together: intentional camera movement, and unwanted shake. Stabilisation separates them.

flowchart TD
  A["Track features<br/>across frames"] --> B["Estimate inter-frame<br/>transform"]
  B --> C["Accumulate into a<br/>camera trajectory"]
  C --> D["Smooth the trajectory<br/>low-pass filter"]
  D --> E["Compensation =<br/>smoothed − original"]
  E --> F["Warp each frame"]
  F --> G["Crop to remove<br/>empty borders"]
  style G fill:#fee,stroke:#c00

Concretely, with horizontal camera position tracked over eight frames:

The unavoidable cost: warping a frame leaves empty regions at the edges, so the output must be cropped. Stronger smoothing means larger corrections means a bigger crop. Typical loss is 5–15% of each dimension, and there is no way around it other than generating the missing borders, which is a guess.

Video denoising

This is where video’s advantage is starkest, and the reason is a single statistical fact.

Sensor noise is independent between frames. The signal is not — it is nearly the same frame to frame. So averaging N aligned frames reduces the noise standard deviation by sqrt(N) while leaving the signal alone.

Nothing an image denoiser does comes close to a free 6 dB. A single-image denoiser must infer what is noise and what is texture, and it gets that wrong on fine detail — which is why denoised photos look plastic. Temporal averaging needs no such inference.

The catch is in the last column. If anything in the scene moves, naive averaging smears it. So real video denoisers align first:

1. Estimate flow from each neighbouring frame to the reference frame
2. Warp neighbours onto the reference using that flow
3. Average, but reject any pixel whose warped value differs too much
   from the reference — that means the alignment failed
4. Fall back to spatial denoising for rejected pixels

Step 3 is what makes it work in practice. Alignment fails at occlusions and on fast motion, and blindly averaging a misaligned pixel produces a ghost far worse than the noise you were removing.

Detection that uses time

Back to the opening problem. Running a full detector on every frame is wasteful. Here is the structure that fixes it.

flowchart LR
  F1["Frame 1<br/>KEYFRAME"] -->|"full detector<br/>expensive"| D1["Detections"]
  D1 -->|"cheap propagation"| D2["Frame 2"]
  D2 -->|"cheap propagation"| D3["Frame 3"]
  D3 -->|"cheap propagation"| D4["Frame 4"]
  F5["Frame 5<br/>KEYFRAME"] -->|"full detector"| D5["Detections"]
  D5 -->|"backward propagation"| D4
  style F1 fill:#fee
  style F5 fill:#fee
  style D2 fill:#efe
  style D3 fill:#efe
  style D4 fill:#efe

Two things make this better than it sounds.

Propagation is genuinely cheap. Moving a box from one frame to the next using flow is a handful of arithmetic operations, against tens of GFLOPs for a detector forward pass.

Propagation runs both directions. Frame 4 is reached from keyframe 5 going backwards as well as from keyframe 1 going forwards. Backward propagation from a nearby keyframe is far more accurate than forward propagation from a distant one, so interior frames are not simply the worst case.

Temporal information also fixes the flicker problem directly. Detections that survive across frames are more trustworthy than one-frame detections:

history = {}   # track_id → list of recent confidences

def stabilise(track_id, conf, window=5, threshold=0.5):
    h = history.setdefault(track_id, [])
    h.append(conf)
    if len(h) > window:
        h.pop(0)
    # Report only if the object was confident in most recent frames
    return sum(c > threshold for c in h) >= len(h) // 2 + 1

This one change removes most of the flicker that per-frame detection produces, and it costs nothing.

Two harder video tasks

Worth knowing about, because they show where the field goes next.

Person search from a single portrait

Given one photograph of a person, find every appearance of them in hours of footage. It is not face recognition — the face is often not visible — and it is not tracking, because appearances are separated by long gaps.

This is a good moment to note that this capability is exactly the one covered in responsible computer vision. It is technically interesting and it is also the core of a surveillance system, so deploying it is a decision about more than accuracy.

Story understanding

Given a full film, answer “why did that character do that?” This needs identity tracking across scenes, dialogue, causal reasoning over long time spans, and knowledge the video never states. Current models handle clips of seconds to minutes. Feature-length narrative reasoning is open.

Putting it together

import cv2
import numpy as np

cap = cv2.VideoCapture("clip.mp4")
KEYFRAME_EVERY = 5

prev_gray, tracks, idx = None, [], 0

while True:
    ok, frame = cap.read()
    if not ok:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    if idx % KEYFRAME_EVERY == 0:
        tracks = run_detector(frame)          # expensive path
    elif prev_gray is not None:
        flow = cv2.calcOpticalFlowFarneback(
            prev_gray, gray, None, 0.5, 3, 15, 3, 5, 1.2, 0)
        tracks = propagate(tracks, flow)      # cheap path

    prev_gray = gray
    idx += 1

def propagate(boxes, flow):
    out = []
    for (x, y, w, h, cls, conf) in boxes:
        # Median flow inside the box is robust to a few bad vectors
        region = flow[y:y+h, x:x+w]
        if region.size == 0:
            continue
        dx = float(np.median(region[..., 0]))
        dy = float(np.median(region[..., 1]))
        # Decay confidence with distance from the keyframe
        out.append((int(x+dx), int(y+dy), w, h, cls, conf * 0.97))
    return out

Two details carry most of the value. Using the median of the flow inside the box, not the mean, means a handful of wild vectors at the box edge cannot drag the whole box away. And decaying confidence on each propagation step means a stale track fades out on its own rather than persisting forever after the object has left.

Basic stabilisation:

transforms = []
prev_pts = cv2.goodFeaturesToTrack(prev_gray, 200, 0.01, 30)

for each frame:
    pts, status, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, prev_pts, None)
    good_prev = prev_pts[status == 1]
    good_new  = pts[status == 1]
    m, _ = cv2.estimateAffinePartial2D(good_prev, good_new)
    transforms.append((m[0, 2], m[1, 2], np.arctan2(m[1, 0], m[0, 0])))

trajectory = np.cumsum(transforms, axis=0)
smoothed = moving_average(trajectory, window=30)
correction = smoothed - trajectory      # apply this as a warp per frame

Practice task

About an hour.

1. Take a 10-second clip. Run a detector on every frame and record the total time and the boxes per frame.
2. Now run it every 5th frame with flow propagation between. Compare total time and compare the boxes on the intermediate frames against the per-frame result. Compute how many you lost.
3. Plot detection confidence for one object over time in both versions. The propagated version should be visibly smoother — that smoothness is the flicker you removed.
4. Add temporal smoothing to the per-frame version using the snippet above. Count how many one-frame false positives it eliminates.
5. Take a shaky handheld clip. Track features, plot the raw trajectory against a 30-frame moving average, and measure how much crop the largest correction demands.
6. Shoot a static scene in low light. Average 8 aligned frames and compare noise against a single frame. Then average 8 frames of a scene with someone walking through it, without alignment, and see the smear.

Step 6 shows both halves of temporal denoising in five minutes.

Summary

Video’s defining property is that consecutive frames are nearly identical, and every technique here exploits that in one of two ways. Redundancy means computation can be reused: run the expensive detector on keyframes and propagate cheaply between them, for roughly a quarter of the compute at a couple of percent accuracy cost. Change means information: differences between frames reveal motion, boundaries and depth that no single frame contains.

Learned flow networks replaced hand-designed optimisation, and FlowNetCorr’s correlation layer illustrates a broader principle — encoding known structure in the architecture beats hoping the network finds it. Interpolation warps along flow rather than averaging, because averaging always ghosts, and it needs a generative step for pixels that neither input frame contains. Stabilisation separates intended camera motion from shake and always costs a crop. Denoising gets a sqrt(N) noise reduction for free from temporal averaging, provided you align first and reject pixels where alignment failed.

The single cheapest improvement to any per-frame system is temporal confidence smoothing. It removes most flicker and costs nothing.

What comes next

Real-time computer vision systems takes these ideas and turns them into a system with a latency budget, covering pipeline design, frame dropping, and where the time actually goes.

If the person-search discussion raised questions, responsible computer vision deals with the part that is not a technical problem.