Real-Time CV Systems: Tracking, Latency, Streaming

A batch job that takes four hours instead of three is mildly annoying. A live camera pipeline that falls 10% behind is a completely different kind of problem: the deficit never clears. After a minute you are two seconds behind, after ten minutes you are twenty, and the vehicle count you publish describes a junction as it was some time ago.

Prerequisites: Optical flow and motion tracking and YOLO object detection.

The job

Eight traffic cameras, 1080p at 30 fps, one GPU. Count vehicles by type and direction, publish counts every second.

$8 \text{ cameras} \times 30 \text{ fps} = 240 \text{ frames per second}$

That number is the whole problem. Everything below is about whether 240 frames per second can be processed, and what to do when it cannot.

The latency budget

Before any code, write down where the time goes.

At 30 fps a frame arrives every:

$\frac{1000 \text{ ms}}{30} = 33.33 \text{ ms}$

This chart is the most useful artefact in the project. It tells you that shaving 20% off inference buys 3.6 ms, more than the entire tracker and publish stages combined.

One GPU, eight streams

The budget above was for one stream. The GPU is shared.

Not every stage costs the same on every frame if you detect less often. Separate the fixed per-frame cost from the detection cost:

Now compute GPU time per stream per second under different detection intervals:

Detect every frame: $30 \times (6.0 + 23.0) = 30 \times 29.0 = 870 \text{ ms per second}$ $\text{streams per GPU} = \frac{1000}{870} = 1.15$

Detect every 5th frame: $6 \times 29.0 + 24 \times 7.5 = 174 + 180 = 354 \text{ ms per second}$ $\text{streams per GPU} = \frac{1000}{354} = 2.82$

This is the highest-leverage optimisation in most video systems, and it costs nothing but a config change. Eight streams still need three GPUs at N=5, but that is far better than seven.

Throughput is not latency

Two numbers, often confused, that move in opposite directions when you batch.

The reason batching helps at all is that a GPU running batch 1 is mostly idle — kernel launch overhead and memory transfers dominate, and the arithmetic units are underused. Batching amortises that fixed cost.

Since our 8 streams are independent, we can batch across streams rather than delaying frames from one stream. That is the good kind of batching: the batch fills in about 4 ms because frames are arriving from 8 sources at once, so the latency cost is small and the throughput gain is real.

Percentiles, not averages

Measured over an hour on one stream:

The tail usually comes from garbage collection, memory allocation, another process on the GPU, or a key frame in the video stream that is more expensive to decode. Find the cause of the p99 rather than trying to shave the median.

When you cannot keep up

Arrival rate 30 fps, service rate 28 fps. The deficit is 2 frames per second, and it accumulates.

For live video, drop. A frame from four seconds ago has no value, and holding it costs memory and delays every frame behind it.

import queue, threading

frames = queue.Queue(maxsize=2)     # tiny on purpose

def capture(cap):
    while True:
        ok, frame = cap.read()
        if not ok: break
        try:
            frames.put_nowait(frame)
        except queue.Full:
            try: frames.get_nowait()   # discard the oldest
            except queue.Empty: pass
            frames.put_nowait(frame)   # keep the newest

The maxsize=2 is the important line. A large queue does not prevent the problem, it hides it — the system appears to keep up for several minutes while quietly building a backlog, then reports results that are minutes stale with no error anywhere.

flowchart LR
  A["Camera<br/>30 fps"] --> B["Queue<br/>maxsize=2"]
  B -->|"full"| X["Drop oldest<br/>increment counter"]
  B --> C["Batch across streams"]
  C --> D["GPU inference"]
  D --> E["NMS + decode"]
  E --> F["Tracker"]
  F --> G["Counting logic"]
  G --> H["Publish"]
  X -.->|"drop rate > 5%"| I["Alert"]

Little’s Law

A useful sanity check. In a stable system:

$L = \lambda W$

where $L$ is the number of items in the system, $\lambda$ the arrival rate, and $W$ the time each spends inside.

With 8 streams at 30 fps and 30.5 ms per frame:

$\lambda = 240 \text{ frames/s}, \qquad W = 0.0305 \text{ s}$ $L = 240 \times 0.0305 = 7.32 \text{ frames}$

So roughly 7 frames should be in flight at any moment. If your monitoring shows 400, the system is not stable and the queue is absorbing a deficit you have not noticed yet.

Tracking between detections

The tracker is what makes N=5 work. The cheapest useful version matches detections to existing tracks by IoU.

A track that misses several consecutive detections is deleted. Set that threshold against the detection interval: at N=5 detections come 167 ms apart, so allowing 3 misses tolerates half a second of occlusion.

ByteTrack’s contribution is worth knowing because it is counterintuitive: it keeps the low-confidence detections that everyone else discards, and uses them in a second matching pass against tracks that failed to match in the first. A partially occluded car often scores 0.3 rather than 0.7, and that low-confidence box is exactly what is needed to keep its ID alive.

Practice task

Use any RTSP stream or a looping video file.

1. Instrument each stage with timers. Build the budget table for your own hardware.
2. Compute your frame budget from the source FPS and compare.
3. Log per-frame latency for 10 minutes and compute p50, p95, p99, and max. Note the gap.
4. Set the queue to maxsize=1000 and run a deliberately slow model. Watch the result age grow.
5. Change it to maxsize=2 with drop-oldest. Watch the drop counter instead.
6. Measure inference at batch 1, 4, 8, 16. Plot throughput and latency together.
7. Implement detect-every-N with a simple IoU tracker. Measure capacity and counting accuracy for N in 1, 3, 5, 10.
8. Apply Little’s Law to your running system and check the predicted in-flight count against reality.

Step 4 is the one worth doing badly on purpose. Seeing the system report cheerful, well-formed results that are two minutes stale, with no error and no warning, is the fastest way to understand why the queue must be small.

Summary

Eight 1080p streams at 30 fps is 240 frames per second, and the per-frame budget is 33.33 ms. Ours summed to 30.5 ms, leaving 8.5% headroom, with inference at 59% — the only stage worth optimising first.

Detect-and-track changed capacity more than anything else: every frame gave 1.15 streams per GPU, every 5th gave 2.82, for 1.3 points of counting accuracy. Past N=5 accuracy collapsed to 88.6%.

Batching raised throughput from 55.6 to 117.6 images per second at batch 8, while latency went from 18 ms to 68 ms. Batching across streams instead of within one gets the throughput without most of the latency.

The mean latency of 28.4 ms looked fine while p90 was 34.8 ms, already over budget. A 7% service shortfall put the system 4.3 seconds behind after a minute and 8.5 minutes behind after two hours, with 14,400 frames in memory. Little’s Law says only 7.32 frames should be in flight.

When you cannot keep up, drop the oldest frame and count the drops.

What comes next

The 18 ms inference figure assumed a particular model on a particular GPU. Deploying computer vision models covers how to move that number — ONNX export, TensorRT, and INT8 quantisation with the scale and zero-point arithmetic worked out — and what it costs in accuracy.