Deploying CV Models: ONNX, TensorRT, Edge, and APIs

The traffic counting system needs 18 ms of inference to fit its budget, and it gets that on a desktop GPU. The council wants it in a weatherproof box on a lamppost, powered over Ethernet, with no fan. Same model, one twentieth of the compute.

Prerequisites: Real-time computer vision systems and CNN architectures.

What deployment actually changes

Three things get it back: a faster runtime, lower numeric precision, and graph-level optimisation. They compound.

Step one: leave PyTorch

PyTorch is built for flexibility — every operation dispatches dynamically, which is what makes debugging pleasant and inference slow. An ONNX export freezes the graph so a runtime can rewrite it.

dummy = torch.randn(1, 3, 640, 640, device='cuda')
model.eval()

torch.onnx.export(
    model, dummy, "detector.onnx",
    input_names=["images"], output_names=["output"],
    dynamic_axes={"images": {0: "batch"}, "output": {0: "batch"}},
    opset_version=17,
    do_constant_folding=True)

Then verify the export, because a silently wrong export is the most common deployment bug:

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("detector.onnx", providers=["CUDAExecutionProvider"])
x = np.random.randn(1, 3, 640, 640).astype(np.float32)

torch_out = model(torch.from_numpy(x).cuda()).detach().cpu().numpy()
onnx_out  = sess.run(None, {"images": x})[0]

print(np.abs(torch_out - onnx_out).max())   # want < 1e-4

Step two: TensorRT

The biggest win is layer fusion. A conv → batchnorm → ReLU sequence is three kernel launches and three round trips to memory. TensorRT folds the batchnorm into the convolution’s weights and applies the ReLU inside the same kernel — one launch, one round trip.

flowchart LR
  subgraph "PyTorch: 3 kernels"
  A["Conv2d"] --> B["BatchNorm2d"] --> C["ReLU"]
  end
  subgraph "TensorRT: 1 kernel"
  D["Fused ConvBnReLU"]
  end
  C -.->|"fuse"| D

trtexec --onnx=detector.onnx \
        --saveEngine=detector_int8.plan \
        --int8 \
        --calib=calibration.cache \
        --workspace=4096

The engine file is specific to the GPU model and the TensorRT version. Build it on the target device, not on your laptop, or it will refuse to load.

FP16 is close to a free lunch. On any GPU with tensor cores it is faster, half the size, and the accuracy loss is smaller than the run-to-run variation of retraining. Enable it before considering anything more involved.

How INT8 quantisation works

A float32 number becomes an int8 number through two parameters.

$q = \text{round}\!\left(\frac{x}{s}\right) + z \qquad\qquad x \approx (q - z) \cdot s$

Worked. A tensor’s observed float range is $[-1.0, 3.0]$. Int8 covers $[-128, 127]$.

$s = \frac{x_{\max} - x_{\min}}{q_{\max} - q_{\min}} = \frac{3.0 - (-1.0)}{127 - (-128)} = \frac{4.0}{255} = 0.0156863$

$z = q_{\min} - \text{round}\!\left(\frac{x_{\min}}{s}\right) = -128 - \text{round}\!\left(\frac{-1.0}{0.0156863}\right) = -128 - (-64) = -64$

Check the endpoints:

$x = -1.0 \;\rightarrow\; \text{round}(-63.75) + (-64) = -64 - 64 = -128 \;\checkmark$ $x = 3.0 \;\rightarrow\; \text{round}(191.25) + (-64) = 191 - 64 = 127 \;\checkmark$ $x = 0.0 \;\rightarrow\; 0 + (-64) = -64 = z \;\checkmark$

Now round-trip some real values:

The error bound is exact and worth remembering:

$\text{max error} = \frac{s}{2} = \frac{0.0156863}{2} = 0.00784$

The outlier problem

Everything above assumed the range was $[-1.0, 3.0]$. Suppose one activation in the tensor hits 47.0.

$s' = \frac{47.0 - (-1.0)}{255} = \frac{48.0}{255} = 0.188235$

$\text{max error}' = \frac{0.188235}{2} = 0.09412$

This is why calibration uses a percentile rather than the true maximum. Clipping 0.01% of values to the boundary is a far better trade than making the other 99.99% twelve times less precise.

Calibration data

INT8 needs to observe real activations to pick ranges. It does not need labels and it does not need many images.

The set must cover your conditions: night, rain, low sun, and heavy traffic. Calibrating only on clear daytime frames sets ranges that clip at night, and the model degrades exactly when it matters.

Cloud or edge

flowchart TD
  A{"Where to run it?"} --> B{"Bandwidth available<br/>and affordable?"}
  B -->|no| E["Edge"]
  B -->|yes| C{"Latency budget<br/>under ~100 ms?"}
  C -->|yes| E
  C -->|no| D{"Can raw video<br/>leave the site?"}
  D -->|no| E
  D -->|yes| F{"Many sites,<br/>frequent model updates?"}
  F -->|yes| G["Cloud"]
  F -->|no| E

Run the numbers for the 8-camera junction.

Bandwidth if video goes to the cloud. 1080p H.264 at 4 Mbps per camera:

$8 \times 4 \text{ Mbps} = 32 \text{ Mbps}$ $32 \times 10^6 \times 86{,}400 \div 8 = 345.6 \text{ GB per day}$ $345.6 \times 365 = 126{,}144 \text{ GB} = 126 \text{ TB per year}$

The bandwidth line is what decides it in practice. Publishing counts instead of video reduces the data by roughly four orders of magnitude — a JSON object of a few hundred bytes once per second is about 26 MB a year per camera.

Making it robust

class Detector:
    def __init__(self, engine_path):
        self.engine = load_engine(engine_path)
        self.warm_up()          # first inference is 10-50x slower

    def warm_up(self):
        dummy = np.zeros((1, 3, 640, 640), dtype=np.float32)
        for _ in range(10):
            self.engine.infer(dummy)

    def predict(self, frame):
        if frame is None or frame.size == 0:
            return []
        try:
            return self.engine.infer(self.preprocess(frame))
        except Exception:
            self.log_error()
            return []           # a missed frame, not a crashed service

The warm-up is not optional. The first inference through a TensorRT engine allocates workspace and loads kernels, and it can take 500 ms. Without warm-up, the first real frame blows the latency budget and the drop counter spikes at every restart.

Drift on outdoor cameras is seasonal and guaranteed. A model calibrated in June meets low winter sun, wet reflective tarmac, and four hours less daylight. Mean detection confidence is the cheapest early warning, because it needs no labels.

Practice task

Take any trained detector or classifier.

1. Export to ONNX. Compare outputs against PyTorch and confirm agreement below 1e-4.
2. Export again without model.eval(). Measure how far the outputs diverge.
3. Time PyTorch, ONNX Runtime, and TensorRT FP32 on the same input. Build the table.
4. Enable FP16. Measure latency and accuracy. Confirm accuracy barely moves.
5. Compute the scale and zero point by hand for one layer’s observed range. Verify with the framework’s values.
6. Calibrate INT8 with 100 real images, then 500, then with random noise. Compare accuracy.
7. Inject one extreme activation into the calibration set. Watch the scale and the accuracy change.
8. Measure the first inference against the hundredth. That gap is why warm-up exists.

Step 7 makes the outlier argument concrete. A single artificial value can move mAP by several points, which is far more dramatic than the explanation suggests.

Summary

The same 25.6M-parameter model went from 142 ms in PyTorch on the Jetson to 4.9 ms in TensorRT INT8 — 29× — for 0.008 mAP. FP16 alone gave 11× for 0.001, which is inside evaluation noise.

INT8 is a scale and a zero point. For a range of $[-1.0, 3.0]$: $s = 4.0/255 = 0.0156863$ and $z = -64$, verified at all three endpoints. Maximum error is $s/2 = 0.00784$.

One outlier at 47.0 pushed the scale to 0.188235 and the error to 0.09412 — twelve times worse for every ordinary value, costing 0.075 mAP. Calibrate on a percentile, not the true maximum.

One hundred real production images beat 500 out-of-domain ones for calibration. Random noise was worthless.

Streaming eight cameras to the cloud is 126 TB a year and £30,510 over three years. The edge box is £1,500 and no video leaves the site.

What comes next

The system now runs fast, cheaply, and on a lamppost. That raises questions that have nothing to do with latency: who is in the frames, who checked whether it works equally well for everyone, and what happens when it is wrong. Responsible computer vision covers privacy, fairness measurement, adversarial robustness, and the release checks worth running before anything goes live.