How Images and Video Are Stored: Colour, JPEG, Frames

Every image your code loads has already been through a lossy transformation that you did not write and probably have not looked at. Most of the time that is fine. Sometimes it is the reason your model is 4% worse than it should be, and you will never find it by tuning hyperparameters.

Prerequisites: image fundamentals. You should know what a pixel and a histogram are.

A problem that is really a storage problem

A site has 12 cameras at 1080p, 25 frames a second, recording continuously. The team budgeted 30 days of retention. The disks fill in four days.

Someone suggests dropping to 12 fps. Someone else suggests fewer cameras. Both are wrong answers, because nobody has looked at what the encoder is doing. The actual fix — changing the keyframe interval and enabling B-frames on the archive stream while leaving the live analytics stream alone — costs nothing and gets them past 30 days.

To make that decision you need to know how images and video are actually stored. That is this post.

An image is a 2D discrete signal

Here is an 8×8 grayscale patch, exactly as the sensor hands it over. The upper rows are bright, and something dark occupies the bottom right.

Two independent decisions were made before you got this.

Decision 1: sampling — how many positions

The sensor divides the scene into a grid and records one number per cell. More cells means more spatial detail. This is resolution, and it is the one everybody already thinks about.

The thing worth knowing is that a feature smaller than one pixel is not blurred — it is gone. If a crack in a component is 0.3 mm wide and your camera resolves 0.8 mm per pixel, no algorithm recovers that crack. This is why the first question on any inspection project is millimetres per pixel, not which model to use.

Decision 2: quantisation — how many levels

Each sample is rounded to one of a fixed number of values. 8-bit gives 256 levels, which is what almost everything uses.

Watch what happens when you reduce it. Take the top-left 4×4 of that patch and requantise:

At 4 bits the numbers are snapping to multiples of 17. At 2 bits, whole regions have collapsed to one value — the entire second row is now identical, and the gentle gradient in row 1 has flattened completely.

That flattening is posterisation Visible banding caused by too few intensity levels, where a smooth gradient becomes a series of flat steps with hard edges between them. , and you can see it in a histogram as a comb: only a few values occur, with gaps between them.

Colour models: four ways to write down a colour

Post 4 covered RGB and HSV. Here is the complete set you will meet, and crucially why each one exists.

RGB and CMYK are opposites in a precise sense. RGB adds light: turn all three up and you get white. CMYK adds ink, and ink absorbs light: pile all of them on and you approach black. That is why your screen and your printer disagree, and why CMYK never appears in computer vision work — nothing you process came from a printer.

YCbCr, and why compression cares

YCbCr is the one that matters here. It splits an RGB pixel into:

• Y — luma, the brightness
• Cb — how blue it is relative to that brightness
• Cr — how red it is relative to that brightness

The conversion is a fixed linear transform:

Y  =  0.299·R + 0.587·G + 0.114·B
Cb = -0.169·R - 0.331·G + 0.500·B + 128
Cr =  0.500·R - 0.419·G - 0.081·B + 128

Take a mid orange, R=200, G=120, B=60:

Y  = 0.299(200) + 0.587(120) + 0.114(60)
   = 59.8 + 70.4 + 6.8  = 137.0
Cb = -0.169(200) - 0.331(120) + 0.500(60) + 128
   = -33.8 - 39.7 + 30.0 + 128 = 84.5
Cr = 0.500(200) - 0.419(120) - 0.081(60) + 128
   = 100.0 - 50.3 - 4.9 + 128 = 172.8

Note the weights on Y: green contributes 0.587 and blue only 0.114. Human vision is far more sensitive to green light, so green carries most of the perceived brightness.

Now the payoff. Your eyes resolve fine detail in brightness much better than in colour. So a codec can store Y at full resolution and store Cb and Cr at half resolution in each direction, and you will not see it.

That is chroma subsampling Storing the colour channels at lower spatial resolution than the brightness channel. 4:2:0 halves the colour resolution both horizontally and vertically. and it is free compression before any clever maths happens:

JPEG: what actually happens to your image

Here is the whole pipeline. It is worth knowing because every step has a consequence you can observe.

flowchart LR
  A["Raw RGB"] --> B["Colour transform<br/>RGB → YCbCr"]
  B --> C["Chroma<br/>downsampling<br/>4:2:0"]
  C --> D["Split into<br/>8×8 blocks"]
  D --> E["Forward DCT<br/>per block"]
  E --> F["Quantisation<br/>← the lossy step"]
  F --> G["Entropy encoding<br/>Huffman"]
  G --> H["JPEG file"]
  style F fill:#fee,stroke:#c00

Decoding runs the same chain backwards: entropy decode, de-quantise, inverse DCT, upsample chroma, convert back to RGB.

Only one step actually destroys information in a tunable way, and it is quantisation. Notice that the DCT itself is not lossy — it is a reversible change of representation. It exists to make the quantisation step effective.

The DCT, concretely

The 64 patterns are fixed and known to both encoder and decoder, so only the 64 weights need storing. The top-left pattern is flat (the block’s average). Moving right adds horizontal stripes of increasing frequency; moving down adds vertical stripes.

Here is why this matters. Take a real 8×8 block from a smooth region of a photo and look at its DCT coefficients:

Look at the size of the numbers. The top-left coefficient is 612. By the bottom-right corner they are 0, 1 and 2.

This is energy compaction The tendency of the DCT to concentrate most of a natural image block's information into a few low-frequency coefficients, leaving the high-frequency ones near zero. , and it is the entire reason JPEG works. Natural images are mostly smooth, so the high-frequency coefficients are almost always tiny.

Quantisation divides each coefficient by a value from a quantisation table and rounds. The table uses large divisors for high frequencies — that is the whole trick. Divide a coefficient of 2 by 40 and round, and you get 0. Divide 612 by 16 and you get 38, which preserves it well.

Zeros compress to almost nothing, so the file shrinks. The JPEG quality slider is just a multiplier on that table.

Why artifacts are square

Because the image was cut into independent 8×8 blocks, and each block was quantised on its own. Neighbouring blocks make slightly different rounding errors, so their edges no longer line up. At low quality you see the grid.

Choosing a format

The rule that matters: never store a segmentation mask or label image as JPEG. A mask has hard edges and exact integer class values. JPEG will smear class 3 and class 4 into a gradient of 3.2, 3.7, 3.9 along every boundary, and your labels become wrong.

Video: mostly storing what changed

A 1080p frame is 1920 × 1080 × 3 = 6.2 MB raw. At 25 fps that is 155 MB per second, or 13.4 TB per day for one camera. Nobody stores that.

The saving comes from a simple observation: consecutive frames are nearly identical. So store one frame properly, then store only the differences for the frames that follow.

The three frame types

A real stream looks something like I B B P B B P B B P B B I .... The gap between I-frames is the keyframe interval, often 1 to 4 seconds.

The B-frame problem, drawn

Here is the fact that decides live-video architecture.

graph LR
  I1["I₁"] --> B1["B₂"]
  P1["P₄"] --> B1
  I1 --> P1
  B1 --> OUT["Displayed<br/>in order:<br/>I₁ B₂ B₃ P₄"]
  style B1 fill:#fee,stroke:#c00
  style P1 fill:#eef

To decode B₂, you need P₄ — a frame that comes after it in playback order. So the encoder must send P₄ before B₂, and the decoder must buffer.

The consequence: B-frames add latency proportional to how far ahead they reference. For a recorded file nobody notices. For a video call or a live analytics feed it is unacceptable, which is why real-time profiles disable B-frames entirely.

Back to the 12-camera problem

Now the fix from the opening is obvious. Run two streams:

• Live analytics stream — no B-frames, short keyframe interval, moderate quality. Latency matters, size does not, because it is not retained.
• Archive stream — B-frames on, keyframe interval at 4 seconds, higher compression. Latency is irrelevant, size is everything.

Neither the frame rate nor the camera count had to change.

Reading this from Python

import cv2

img = cv2.imread("photo.jpg")           # BGR uint8, already decoded
print(img.shape, img.dtype)             # (1080, 1920, 3) uint8

# Convert to YCbCr and look at the channels separately
ycc = cv2.cvtColor(img, cv2.COLOR_BGR2YCrCb)
y, cr, cb = cv2.split(ycc)
print("Y  range:", y.min(), y.max())
print("Cr range:", cr.min(), cr.max())

# Control JPEG quality explicitly. Default is 95; do not go below 90 for data.
cv2.imwrite("out.jpg", img, [cv2.IMWRITE_JPEG_QUALITY, 95])

# Masks must be lossless
cv2.imwrite("mask.png", mask)           # correct
# cv2.imwrite("mask.jpg", mask)         # wrong: smears class boundaries

Checking whether an image has been heavily compressed:

import numpy as np

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
levels = np.unique(gray).size
print(f"{levels} distinct levels of 256")
# Well under 200 on a normal photo suggests requantisation somewhere upstream.

Reading video without silently landing on keyframes:

cap = cv2.VideoCapture("clip.mp4")
fps = cap.get(cv2.CAP_PROP_FPS)

frames, idx = [], 0
while True:
    ok, frame = cap.read()      # sequential read: no keyframe snapping
    if not ok:
        break
    if idx % int(fps) == 0:     # one frame per second
        frames.append(frame)
    idx += 1
cap.release()
print(f"{len(frames)} frames from {idx} total at {fps} fps")

Reading sequentially and discarding is slower than seeking, and it is correct. Seeking with CAP_PROP_POS_FRAMES is where the keyframe-snapping bias comes from.

Practice task

About forty minutes, and it will change how you handle data.

1. Take one photo you own. Save it at JPEG quality 95, 75, 50 and 20. Record the four file sizes.
2. Load the quality-20 version, zoom into a smooth area such as sky or a plain wall, and find the 8×8 grid.
3. Count distinct grey levels in the original and in the quality-20 version using the snippet above.
4. Now the important one. Take the quality-75 file, load it, save it again at 75, and repeat ten times. Compare round 1 against round 10 side by side.
5. Finally, take a short video. Extract frames sequentially, then extract them again using CAP_PROP_POS_FRAMES seeking. Compare how many unique frames each method actually produced.

Step 4 is the one worth doing. The degradation from repeated encoding is much larger than most people expect, and it is exactly what a careless dataset pipeline does.

Summary

An image is sampled in space and quantised in value, and both choices throw information away permanently. Colour models exist for different jobs: RGB for screens, CMYK for ink, HSV for selecting colours under changing light, and YCbCr for compression, because separating brightness from colour lets a codec keep brightness sharp and blur the colour.

JPEG converts to YCbCr, halves the colour resolution, splits into 8×8 blocks, transforms each to frequencies, and rounds the high frequencies to zero. That is where both the compression and the square artifacts come from. Video goes further and stores differences: I-frames stand alone, P-frames reference the past, and B-frames reference both past and future — which is why they compress best and why they cannot be used live.

The practical consequences are worth more than the theory. Never JPEG a mask. Never re-encode a dataset repeatedly. Check for comb histograms. And read video sequentially unless you have a reason not to.

What comes next

Now that you know what has already been done to your pixels, OpenCV setup and first 10 tasks gets you loading and manipulating them in code.

If you want to keep going on the image-processing side, intensity transforms and frequency-domain filtering picks up the frequency idea introduced here and uses it to actually filter images — the same domain the DCT works in, applied to enhancement rather than compression.