Image Fundamentals: Color, Histograms, Noise, Filtering

Most vision projects fail on image quality long before they fail on modelling. This post covers the three things that decide image quality: how colour is stored, what a histogram tells you, and how to remove noise without destroying the detail you need.

Prerequisites: Math for CV beginners for convolution and array shapes.

A problem where colour choice decides everything

A packing shed sorts tomatoes on a conveyor. Ripe tomatoes are red and go to the fresh crate. Unripe ones are green and go back for another day. A camera above the belt takes one photo per tomato.

The shed runs from 6am to 8pm. At 6am the light is dim. At noon a skylight floods the belt. At 8pm the overhead lamps are the only source. The tomatoes do not change. The light changes constantly.

This is the classic case where the colour space you pick decides whether the system works.

How colour is stored

An image with colour is three grids stacked. Each grid is one channel One of the parallel 2D grids that make up a colour image. RGB has three: one for red, one for green, one for blue. .

OpenCV squeezes hue into 0–179 instead of 0–359 so it fits in a uint8. Every hue value in OpenCV is half the degree value. Forgetting that is the most common HSV bug.

Additive and subtractive: why CMYK exists

One colour model is missing from that table, and it is worth a paragraph because the contrast explains what RGB actually is.

RGB is additive. Each channel says how much light of that colour to emit. Turn all three to maximum and the light adds up to white. Turn them all off and you get black. This is how every screen and every camera sensor works, which is why RGB is the default everywhere in computer vision.

CMYK is subtractive. Cyan, magenta, yellow and black describe how much ink to lay down, and ink does not emit light — it absorbs it. Pile all of them on and almost nothing reflects back, so you approach black. Leave the paper bare and you get white.

You will not process CMYK images, because nothing you capture comes from a printer. Knowing it exists matters for one practical reason: it is why a colour that looked right on your screen prints differently, and why design assets sometimes arrive in a colour space your loader silently mangles. If cv2.imread gives you strange colours on a file from a designer, check whether the file is CMYK.

Why RGB breaks and HSV does not

Take one specific red tomato. Here are its pixel values photographed at three times of day.

A rule like “red channel above 150” works at noon, half-works in the morning, and fails completely in the evening. But look closer: every channel dropped by roughly the same proportion. The ratios between R, G, and B barely moved. The colour did not change; only the amount of light did.

HSV is built to expose exactly that. HSV Hue, Saturation, Value. Hue is which colour it is, saturation is how vivid, and value is how bright. Changing the light mostly changes value alone. splits the “which colour” part away from the “how bright” part.

Converting RGB to HSV by hand

Take the noon pixel: $R = 200$, $G = 40$, $B = 35$.

Step 1: scale to 0–1.

$r = \frac{200}{255} = 0.784, \quad g = \frac{40}{255} = 0.157, \quad b = \frac{35}{255} = 0.137$

Step 2: find max, min, and the spread.

$\text{max} = 0.784 \;(\text{from } r), \quad \text{min} = 0.137, \quad \Delta = 0.784 - 0.137 = 0.647$

Step 3: Value is just the max.

$V = 0.784 \quad \rightarrow \quad 0.784 \times 255 = 200 \text{ (OpenCV scale)}$

Step 4: Saturation is the spread relative to the max.

$S = \frac{\Delta}{\text{max}} = \frac{0.647}{0.784} = 0.825 \quad \rightarrow \quad 0.825 \times 255 = 210$

Step 5: Hue depends on which channel was the max. Since red won:

$H = 60° \times \frac{g - b}{\Delta} = 60° \times \frac{0.157 - 0.137}{0.647} = 60° \times 0.031 = 1.86°$

OpenCV halves it: $H_{cv} = 0.93 \approx 1$.

Now run the identical steps on the evening pixel ($R=90$, $G=18$, $B=16$):

$r = 0.353, \; g = 0.071, \; b = 0.063, \quad \text{max} = 0.353, \; \Delta = 0.290$ $V = 0.353 \times 255 = 90$ $S = \frac{0.290}{0.353} = 0.822 \times 255 = 210$ $H = 60° \times \frac{0.071 - 0.063}{0.290} = 60° \times 0.027 = 1.63° \rightarrow H_{cv} \approx 1$

Here is the whole point of this post in one table:

All three channels drop with the light. Hue does not.

Now do the same for an unripe green tomato: $R = 80$, $G = 150$, $B = 60$.

$r = 0.314, \; g = 0.588, \; b = 0.235, \quad \text{max} = 0.588 \;(g), \; \Delta = 0.353$

When green is the max, the hue formula shifts by 2:

$H = 60° \times \left( \frac{b - r}{\Delta} + 2 \right) = 60° \times \left( \frac{0.235 - 0.314}{0.353} + 2 \right) = 60° \times 1.776 = 106.6°$

$H_{cv} = 53$

So ripe sits at hue 1 and unripe sits at hue 53. A single number separates them, and that number does not care what time of day it is.

import cv2
import numpy as np

bgr = cv2.imread("tomato.jpg")
hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)

# red wraps around 0 in hue, so it needs two ranges
lower_red_1 = np.array([0,   120, 60])
upper_red_1 = np.array([10,  255, 255])
lower_red_2 = np.array([170, 120, 60])
upper_red_2 = np.array([179, 255, 255])

mask = cv2.inRange(hsv, lower_red_1, upper_red_1) | cv2.inRange(hsv, lower_red_2, upper_red_2)
ripe_fraction = mask.sum() / 255 / mask.size
print(f"ripe pixels: {ripe_fraction:.1%}")

Note the V lower bound of 60 in the mask. Very dark pixels have unreliable hue, because $\Delta$ is tiny and dividing by a tiny number amplifies noise. Always floor the value channel when you build a hue mask.

Which colour space when

graph TD
  Q{"What does your rule<br/>depend on?"} -->|"shape, edges, texture"| G["Grayscale<br/>simpler and faster"]
  Q -->|"which colour it is"| H["HSV<br/>threshold on hue"]
  Q -->|"how similar two colours look"| L["LAB<br/>Euclidean distance is meaningful"]
  Q -->|"feeding a neural network"| R["RGB, normalized<br/>the model learns the rest"]

Histograms: reading an image in one glance

A histogram A count of how many pixels fall into each brightness range. It ignores where the pixels are and only records how many of each value exist. is the fastest diagnostic tool in computer vision. Before you debug an algorithm, look at the histogram.

Compute one by hand

Take this 4×4 grayscale patch:

Use four bins, each 64 values wide:

Plotted, those four counts look like this:

Two tall peaks with a completely empty gap between them. That shape is called bimodal A histogram with two clear peaks, meaning the image contains two distinct brightness populations such as an object and its background. , and it is the best news you can get: any cutoff between 64 and 191 splits object from background perfectly. This is exactly the condition Otsu thresholding is designed to find automatically.

What a histogram cannot tell you

Before you trust histograms too much, here is the limitation, and it is a large one.

These three 4×4 images look nothing alike:

Count the values in each. Every one has exactly eight pixels at 10 and eight at 200. Their histograms are identical:

This matters practically. A histogram tells you whether an image is exposed correctly, whether it is bimodal enough to threshold, and whether it has been requantised. It tells you nothing about whether the object is one blob or scattered noise, nothing about texture, and nothing about shape. Two images with the same histogram can require completely different processing.

What real histograms look like

import cv2
import matplotlib.pyplot as plt

gray = cv2.imread("frame.jpg", cv2.IMREAD_GRAYSCALE)
hist = cv2.calcHist([gray], [0], None, [256], [0, 256])

print("clipped at 0:", hist[0][0], " clipped at 255:", hist[255][0])
plt.plot(hist)
plt.xlabel("Pixel value"); plt.ylabel("Count"); plt.show()

Those two clipping counts are worth printing every time. If thousands of pixels sit at exactly 255, the sensor saturated and that information is gone forever. No filter, no model, no amount of training data brings it back. Fix the camera.

Brightness and contrast are two different operations

People use these words loosely, but on a histogram they do completely different things. Brightness slides the histogram. Contrast stretches it.

Take five pixels from a dull, flat image: 100, 105, 110, 115, 120. The mean is 110 and the spread is small.

$\sigma = \sqrt{\frac{(-10)^2 + (-5)^2 + 0^2 + 5^2 + 10^2}{5}} = \sqrt{\frac{250}{5}} = \sqrt{50} = 7.071$

Brightness: add a constant. Add 40 to every pixel:

$100, 105, 110, 115, 120 \;\rightarrow\; 140, 145, 150, 155, 160$

The mean moved from 110 to 150. The spread is unchanged, still 7.071, because adding the same number to everything moves every value equally. The image is brighter and just as flat as before.

Contrast: multiply around a pivot. Multiply the distance from the mean by 3:

$\text{new} = 3(v - 110) + 110$

$100 \rightarrow 80, \quad 105 \rightarrow 95, \quad 110 \rightarrow 110, \quad 115 \rightarrow 125, \quad 120 \rightarrow 140$

The mean is still 110, but the spread tripled to 21.21. The values now use 60 levels instead of 20. That is contrast.

Notice the orange histogram is exactly the same shape as the grey one, just relocated. If your image looks washed out, adding brightness will not help. You need the green operation.

OpenCV does both with one call, using $\text{new} = \alpha \cdot v + \beta$:

brighter  = cv2.convertScaleAbs(gray, alpha=1.0, beta=40)     # slide  +40
contrasty = cv2.convertScaleAbs(gray, alpha=3.0, beta=-220)   # stretch x3 about 110

print(gray.mean(), gray.std())            # 110.0  7.07
print(contrasty.mean(), contrasty.std())  # 110.0  21.21

The $\beta = -220$ comes from keeping the pivot fixed: to leave 110 unmoved while tripling, you need $110 - 3(110) = -220$.

Contrast has a hard limit: multiply too aggressively and values run past 255, where OpenCV clamps them. With $\alpha = 3$, any original pixel above 158 lands beyond 255 and gets flattened into a solid white blob. That is why the automatic version below scales by the actual distribution instead of a number you guessed.

Dynamic range is a third thing

Brightness and contrast are two operations. There is a third quantity people conflate with both, and separating it prevents a whole category of wasted effort.

Contrast is about the spread of values around the middle. Dynamic range is about the ratio between the extremes. A thermal sensor reading from 3 to 4000 has a dynamic range of about 1300:1, which no 8-bit display can show linearly — squeeze it into 0–255 and everything below 200 collapses into near-black.

Contrast stretching does not fix that, because the problem is not that values are bunched together. They are spread across three orders of magnitude, and 256 levels cannot represent three orders of magnitude with detail at both ends. The fix is a non-linear curve such as a log transform, which is covered in intensity transforms and frequency-domain filtering.

The related symptom worth recognising here is the opposite problem — too few distinct levels:

If you see that comb pattern in your data, the file has been through a lossy step: heavy JPEG compression, an 8-bit save of 16-bit data, or a screenshot of a screenshot. It is worth finding the original before you train on it, and how images and video are stored covers why it happens.

import numpy as np
levels = np.unique(gray).size
print(f"{levels} distinct levels of 256")   # well under 200 is suspicious

Histogram equalization, worked out

Equalization spreads a bunched histogram across the full range. It does this by mapping each level through the cumulative distribution function The running total of pixel counts up to and including each brightness level. Levels with many pixels get spread further apart. of the histogram.

Take a tiny 16-pixel image using 3-bit levels (0 to 7). Suppose the counts are:

Every pixel sits between 3 and 6, so the image looks flat and grey. The mapping formula is:

$\text{new level} = \text{round}\left( \frac{\text{CDF}(v) - \text{CDF}_{min}}{N - \text{CDF}_{min}} \times (L - 1) \right)$

With $N = 16$, $\text{CDF}_{min} = 3$, and $L = 8$:

$\text{level } 3 \rightarrow \text{round}\left(\frac{3-3}{13} \times 7\right) = 0$ $\text{level } 4 \rightarrow \text{round}\left(\frac{8-3}{13} \times 7\right) = \text{round}(2.69) = 3$ $\text{level } 5 \rightarrow \text{round}\left(\frac{12-3}{13} \times 7\right) = \text{round}(4.85) = 5$ $\text{level } 6 \rightarrow \text{round}\left(\frac{16-3}{13} \times 7\right) = 7$

The values that spanned 3 to 6 now span 0 to 7. Contrast more than doubled, and the levels with the most pixels (level 4, with 5 pixels) got the biggest jump.

That plot shows the key idea. Equalization never changes the counts, only where they sit. Levels with many pixels get pushed far apart, levels with few get squeezed together, and the gap between levels 4 and 5 (5 pixels between them) opened wider than the gap between 5 and 6 (4 pixels).

Here is the same thing seen through the CDF, which is the curve doing the work:

Where the blue CDF is steep, many pixels share that brightness, so equalization spends more of the output range there. Where it is flat, few pixels exist and the output range is not wasted on them. That single rule is the whole algorithm.

equalized = cv2.equalizeHist(gray)                        # whole image at once

clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
local = clahe.apply(gray)                                 # per-tile, gentler

Plain equalizeHist uses one histogram for the whole image, so a bright window in one corner ruins the mapping everywhere else. CLAHE Contrast Limited Adaptive Histogram Equalization. It equalizes small tiles independently and caps how much any one level can be stretched, which avoids amplifying noise. divides the image into tiles and equalizes each one, with a cap on how aggressive it can be. In practice CLAHE is what you reach for.

Noise and the filters that remove it

Noise is unwanted variation in pixel values. Two types cover most of what you meet.

Gaussian noise is small random wobble added to every pixel, caused by sensor electronics. It gets much worse in low light and at high ISO. A pixel whose true value is 50 might read 47, 53, 48, 52 across frames.

Salt-and-pepper noise is a small number of pixels set to extreme values, caused by dead sensor elements, dust, or transmission errors. A pixel whose true value is 50 reads 0 or 255.

They need different treatment, and the reason is arithmetic.

Their histograms make the difference obvious. Take a flat patch whose true value is 50, and corrupt it two ways:

The blue hill has no pixels far from 50, so averaging works well: the errors sit on both sides and cancel out. The red histogram is mostly correct, but the two short bars at 0 and 255 are as far from the truth as it is possible to be. Averaging drags the answer towards them. Sorting and taking the middle value ignores them completely, because a value’s rank does not care how extreme it is.

That is the whole reason there are two different filters.

One noisy window, two filters

Here is a 3×3 window with one salt pixel. The true value in this region is about 50.

Mean. Sum the nine values: $50 + 52 + 240 + 48 + 50 + 51 + 51 + 49 + 52 = 643$.

$\text{mean} = \frac{643}{9} = 71.4$

Median. Sort them: 48, 49, 50, 50, 51, 51, 52, 52, 240. Take the middle one:

$\text{median} = 51$

One outlier dragged the mean 21 levels away from the truth. The median did not move. That single fact decides which filter you use.

The Gaussian kernel, derived

A plain mean kernel weights all nine pixels equally, which blurs more than necessary. A Gaussian kernel weights the centre more. The weights come from:

$G(x, y) = e^{-\frac{x^2 + y^2}{2\sigma^2}}$

For a 3×3 kernel with $\sigma = 1$, the offsets are $-1, 0, +1$ in each direction:

• Centre, $(0,0)$: $e^{0} = 1.000$
• Edges, $(\pm1, 0)$ and $(0, \pm1)$: $e^{-1/2} = 0.607$
• Corners, $(\pm1, \pm1)$: $e^{-2/2} = 0.368$

Sum of all nine: $1.000 + 4(0.607) + 4(0.368) = 4.898$. Divide each by that sum so the weights add to 1:

The integer version on the right is what you see in textbooks. Dividing by 16 gives 0.0625, 0.125, and 0.25, which is close enough to the exact values and much faster to compute.

The important control is $\sigma$, not the kernel size. Larger $\sigma$ means heavier smoothing. OpenCV picks a sensible kernel size for you if you pass 0:

blurred = cv2.GaussianBlur(gray, (0, 0), sigmaX=1.5)   # size chosen from sigma

Choosing a filter

graph TD
  N{"What kind of noise<br/>do you see?"} -->|"grainy everywhere"| G["Gaussian blur<br/>start with sigma = 1"]
  N -->|"isolated white or<br/>black dots"| M["Median blur<br/>kernel 3 or 5"]
  N -->|"grainy, but edges<br/>must stay crisp"| B["Bilateral filter<br/>slower, worth it"]
  N -->|"none visible"| S["Do not filter.<br/>You only lose detail."]
  G --> C["Then threshold or<br/>detect edges"]
  M --> C
  B --> C

The bilateral filter A blur whose weights depend on both how far a neighbour is and how different its value is. Neighbours across an edge get near-zero weight, so edges survive. is the one worth understanding. A Gaussian blur averages a pixel with everything nearby, including pixels on the other side of an edge, which is exactly why edges soften. A bilateral filter adds a second weight based on value difference:

$w(i, j) = \underbrace{e^{-\frac{d^2}{2\sigma_s^2}}}_{\text{how far}} \times \underbrace{e^{-\frac{(I_i - I_j)^2}{2\sigma_r^2}}}_{\text{how different}}$

If a neighbour is 150 levels brighter, the second term collapses to near zero and that neighbour contributes nothing. The result is smooth regions with sharp boundaries.

den = cv2.bilateralFilter(img, d=9, sigmaColor=75, sigmaSpace=75)

It is roughly ten times slower than a Gaussian blur. Use it when edge sharpness matters and you can afford the time.

Putting it together

Here is the preprocessing chain for the tomato line, in the order it runs.

graph LR
  A["BGR frame<br/>from camera"] --> B["Median blur 3x3<br/>kill dead pixels"]
  B --> C["Convert to HSV"]
  C --> D["inRange on hue<br/>with a V floor"]
  D --> E["Binary mask"]
  E --> F["Count mask pixels<br/>compare to threshold"]
  F --> G["ripe / unripe"]
  G --> H["Log hue histogram<br/>per hour"]
  H -->|"histogram drifts"| C

import cv2
import numpy as np

def classify_tomato(bgr):
    clean = cv2.medianBlur(bgr, 3)
    hsv = cv2.cvtColor(clean, cv2.COLOR_BGR2HSV)

    red = cv2.inRange(hsv, (0, 120, 60), (10, 255, 255)) | \
          cv2.inRange(hsv, (170, 120, 60), (179, 255, 255))
    green = cv2.inRange(hsv, (35, 60, 60), (85, 255, 255))

    red_px, green_px = int(red.sum() // 255), int(green.sum() // 255)
    if red_px + green_px < 500:
        return "no tomato detected"
    return "ripe" if red_px > green_px else "unripe"

Notice the last check. Comparing red pixels against green pixels rather than against a fixed count means the rule works whether the tomato is close to the camera or far from it. Fixed absolute counts break the moment anything about the camera position changes.

Practice task

1. Photograph one coloured object under three lighting conditions: bright daylight, a dim room, and under a coloured lamp.
2. Print the mean BGR and mean HSV of a small central patch in each photo. Confirm that hue moves least.
3. Plot the grayscale histogram for each. Identify which one is under-exposed and check whether any pixels are clipped at 0 or 255.
4. Add salt-and-pepper noise artificially and remove it with both GaussianBlur and medianBlur. Compare the results.

import numpy as np

def add_salt_pepper(gray, amount=0.02):
    out = gray.copy()
    n = int(amount * gray.size)
    ys = np.random.randint(0, gray.shape[0], n)
    xs = np.random.randint(0, gray.shape[1], n)
    out[ys[:n//2], xs[:n//2]] = 0
    out[ys[n//2:], xs[n//2:]] = 255
    return out

Compare the two filtered results side by side. The Gaussian version will still show grey smudges where the noise was. The median version will look as if the noise was never there.

Summary

Colour, contrast, and noise are the three properties that decide whether your algorithm has a chance.

RGB mixes colour and brightness into every channel, so any fixed RGB rule dies when the light changes. HSV separates them, and you saw a hand-worked conversion showing hue holding at 1 while the red channel fell from 200 to 90. RGB is additive and makes white; CMYK is subtractive and makes black, which is why it belongs to printers and never appears in vision work. Histograms tell you in one glance whether an image is under-exposed, over-exposed, clipped, or cleanly bimodal — but they discard every bit of spatial information, so three completely different images can share one histogram. CLAHE fixes low contrast without blowing out bright regions. Noise type decides filter choice: median for isolated outliers, Gaussian for grain, bilateral when edges must stay sharp.

What comes next

How images and video are stored covers what happened to your pixels before your code ever saw them — sampling, bit depth, JPEG, and video frame types. It is the natural companion to this post, because most of the odd histograms you will meet in real data are explained there.

OpenCV setup and your first 10 tasks gets a working environment on your machine and walks through ten exercises that put all of this into practice. After that, edge detection and thresholding uses the histograms from this post to pick thresholds automatically instead of by hand.