Edge Detection and Thresholding That Actually Work

Thresholding and edge detection are the two ways to turn a grayscale image into something you can count and measure. Both look trivial in a tutorial and both fall apart on real photos. This post shows exactly why, and works out the maths that makes them survive.

Prerequisites: Image fundamentals for histograms and filtering.

The problem: a form photographed on a desk

You need to read a printed form. Someone photographs it on a desk with a lamp on the left. The left half of the page is bright white; the right half sits in shadow and is closer to mid-grey.

The text is black in both halves. To a human it reads fine. To a global threshold it is a disaster: any cutoff dark enough to catch the text on the shadowed side turns the entire shadowed background black too.

Here is a row of pixel values sampled straight across that page, crossing four letters:

The red line is a threshold of 128. On the left it separates text (40, 45) from paper (238, 235) perfectly. On the right the paper itself sits at 114–120, below the line. Every background pixel on that side gets classified as ink.

Nothing is wrong with the threshold value. The problem is that one number is being asked to describe two different lighting conditions.

Thresholding, three ways

graph TD
  Q{"Is brightness even<br/>across the image?"} -->|"yes, and fixed forever"| F["Fixed threshold<br/>cv2.threshold with a number"]
  Q -->|"yes, but it drifts<br/>between images"| O["Otsu<br/>picks the number per image"]
  Q -->|"no, it varies<br/>within one image"| A["Adaptive<br/>a different number per region"]
  F --> R["Fastest, most fragile"]
  O --> S["Needs a bimodal histogram"]
  A --> T["Handles shadows and gradients"]

Fixed threshold

_, binary = cv2.threshold(gray, 128, 255, cv2.THRESH_BINARY)

Every pixel above 128 becomes 255, everything else 0. It is the fastest option and the first thing to try when you control the lighting completely. It is also the first thing to break when anything changes.

Otsu: computing the threshold by hand

Otsu's method An automatic threshold that tries every possible cutoff and keeps the one that makes the two resulting groups as different from each other as possible. removes the magic number. Here is the whole algorithm worked out on a small histogram.

Take an image with six grey levels (0 to 5) and these pixel counts:

Total pixels $N = 6 + 10 + 2 + 2 + 10 + 6 = 36$.

For each candidate threshold $t$, split the pixels into “at or below $t$” (group 0) and “above $t$” (group 1). Otsu maximises the between-class variance:

$\sigma_b^2(t) = \omega_0 \, \omega_1 \, (\mu_0 - \mu_1)^2$

where $\omega$ is each group’s share of the pixels and $\mu$ is each group’s mean level.

Try $t = 1$. Group 0 is levels 0 and 1, which is $6 + 10 = 16$ pixels. Group 1 is levels 2–5, which is 20 pixels.

$\omega_0 = \frac{16}{36} = 0.444, \qquad \omega_1 = \frac{20}{36} = 0.556$ $\mu_0 = \frac{(0)(6) + (1)(10)}{16} = \frac{10}{16} = 0.625$ $\mu_1 = \frac{(2)(2) + (3)(2) + (4)(10) + (5)(6)}{20} = \frac{4 + 6 + 40 + 30}{20} = \frac{80}{20} = 4.000$ $\sigma_b^2(1) = (0.444)(0.556)(0.625 - 4.000)^2 = 0.247 \times 11.39 = 2.812$

Try $t = 2$. Group 0 is levels 0–2, which is 18 pixels. Group 1 is levels 3–5, also 18.

$\omega_0 = \omega_1 = 0.500$ $\mu_0 = \frac{0 + 10 + 4}{18} = \frac{14}{18} = 0.778, \qquad \mu_1 = \frac{6 + 40 + 30}{18} = \frac{76}{18} = 4.222$ $\sigma_b^2(2) = (0.5)(0.5)(0.778 - 4.222)^2 = 0.25 \times 11.86 = 2.965$

Do the same for every threshold:

The maximum is at $t = 2$, right in the histogram’s valley. Otsu found it without being told anything about the image.

otsu_val, binary = cv2.threshold(gray, 0, 255,
                                 cv2.THRESH_BINARY + cv2.THRESH_OTSU)
print("Otsu chose:", otsu_val)

The 0 you pass as the threshold is ignored; Otsu computes its own. Print otsu_val across a batch. If it swings between 60 and 190 across images, your lighting is unstable and you should be looking at the camera setup, not the algorithm.

Otsu assumes two peaks. Feed it a smooth single-hill histogram and it still returns a number, but that number is arbitrary and will move unpredictably from image to image.

This is what a histogram with no valley looks like next to one Otsu can actually work with:

The red histogram is the dangerous case, because Otsu still returns a confident-looking number for it. Where the green curve is flat between the peaks, thousands of pixels sit right under the red curve’s split point, and every one of them can flip class from one photo to the next.

Adaptive: a different threshold per region

For the shadowed form, compute a local threshold from each pixel’s neighbourhood instead of the whole image.

adaptive = cv2.adaptiveThreshold(
    gray, 255,
    cv2.ADAPTIVE_THRESH_GAUSSIAN_C,   # weighted local mean
    cv2.THRESH_BINARY,
    blockSize=31,                     # neighbourhood size, must be odd
    C=7,                              # subtracted from the local mean
)

Take the shadowed side of the form. The local mean over a 31×31 window of mostly paper is about 117. With $C = 7$, the local threshold there is:

$T_{local} = 117 - 7 = 110$

Ink pixels at 35 fall below 110 and become black. Paper at 116 stays white. Meanwhile on the lit side the local mean is about 234, so:

$T_{local} = 234 - 7 = 227$

Ink at 40 is still below. The same code produced two thresholds 117 apart, which is exactly what the image needed.

Two parameters to tune, and they do different things:

• blockSize must be larger than the features you want to keep. For 12-pixel-tall text, 31 works. Set it to 5 and the window sits entirely inside a letter stroke, so the letter’s own darkness becomes the local mean and the stroke disappears.
• C is a bias against noise. On a perfectly uniform patch the local mean equals the pixel value, so without C every tiny fluctuation flips a pixel. Values of 5 to 10 are typical.

Comparison

Edge detection

Thresholding classifies pixels by their value. Edge detection looks for places where the value changes. That distinction matters when object and background have similar brightness but a visible boundary between them.

Sobel: gradient size and direction

Sobel applies two kernels, one for horizontal change and one for vertical.

Work one out on a patch containing a diagonal edge, dark in the top-left corner and bright in the bottom-right:

Horizontal gradient. Multiply by Sobel-X and add:

$\text{row 0: } (-1)(10) + (0)(10) + (1)(90) = 80$ $\text{row 1: } (-2)(10) + (0)(90) + (2)(90) = -20 + 180 = 160$ $\text{row 2: } (-1)(90) + (0)(90) + (1)(90) = 0$ $G_x = 80 + 160 + 0 = 240$

Vertical gradient. Multiply by Sobel-Y:

$\text{row 0: } (-1)(10) + (-2)(10) + (-1)(90) = -120$ $\text{row 1: } 0$ $\text{row 2: } (1)(90) + (2)(90) + (1)(90) = 360$ $G_y = -120 + 0 + 360 = 240$

Magnitude — how strong the edge is:

$|G| = \sqrt{G_x^2 + G_y^2} = \sqrt{240^2 + 240^2} = 240\sqrt{2} = 339.4$

Direction — which way brightness increases fastest:

$\theta = \arctan\!\left(\frac{G_y}{G_x}\right) = \arctan\!\left(\frac{240}{240}\right) = 45°$

The gradient points at 45°, down and to the right. The edge itself runs perpendicular to that, at 135°, which matches the diagonal boundary in the patch.

import cv2, numpy as np

gx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=3)    # note: CV_64F, not uint8
gy = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=3)

magnitude = cv2.magnitude(gx, gy)
direction = np.arctan2(gy, gx) * 180 / np.pi        # degrees, -180 to 180

vis = cv2.convertScaleAbs(magnitude)                # rescale to 0-255 for viewing

The cv2.CV_64F matters. Gradients are signed and often exceed 255, as the 339 above shows. Asking for uint8 output clips or wraps the values, and edges in one direction disappear entirely.

Try filters interactively

Select “Sobel X” and step the filter across. Watch where the response is large and where it is zero. The filter never sees the whole object; it only ever reports local change, which is why edge output is a set of lines rather than a filled region.

Canny: turning thick gradients into thin lines

Raw Sobel magnitude is blurry: a one-pixel edge produces a two- or three-pixel-wide bright band. Canny adds three steps to fix that.

graph LR
  A["Grayscale"] --> B["1. Gaussian blur<br/>remove noise"]
  B --> C["2. Sobel<br/>magnitude + direction"]
  C --> D["3. Non-maximum suppression<br/>thin to 1 pixel"]
  D --> E["4. Hysteresis<br/>two thresholds"]
  E --> F["Clean single-pixel edges"]

Step 3, non-maximum suppression. For each pixel, look along its gradient direction and compare it to the two neighbours on either side. Keep it only if it is the largest of the three.

Say three pixels along a gradient direction have magnitudes 210, 339, 180:

339 is the local peak, so it survives. Its neighbours were part of the same blurry band and get zeroed. A three-pixel-wide edge becomes one pixel wide.

Step 4, hysteresis. Two thresholds instead of one. A pixel above the high threshold is definitely an edge. A pixel above the low threshold is an edge only if it connects to a definite one. Anything below the low threshold is discarded.

With low = 50 and high = 150, walk along a chain of edge pixels:

This is what stops a single edge from breaking into dashes wherever it dips slightly in contrast. Pixels B and C would have been lost by any single threshold above 90, taking a chunk out of the middle of a real edge. Pixel E, a noise spike with no neighbours, is correctly dropped even though it is stronger than B.

blurred = cv2.GaussianBlur(gray, (0, 0), 1.4)
edges = cv2.Canny(blurred, threshold1=50, threshold2=150)

A good starting rule is high = 3 × low, or derive both from the image:

def auto_canny(img, sigma=0.33):
    v = np.median(img)
    low = int(max(0, (1.0 - sigma) * v))
    high = int(min(255, (1.0 + sigma) * v))
    return cv2.Canny(img, low, high)

Thresholding or edges?

A pipeline that holds up

import cv2
import numpy as np

def binarize(bgr, mode="auto"):
    gray = cv2.cvtColor(bgr, cv2.COLOR_BGR2GRAY)

    # 1. how uneven is the lighting? compare corner means
    h, w = gray.shape
    k = min(h, w) // 6
    corners = [gray[:k, :k].mean(), gray[:k, -k:].mean(),
               gray[-k:, :k].mean(), gray[-k:, -k:].mean()]
    spread = max(corners) - min(corners)

    # 2. clean noise before any decision
    gray = cv2.medianBlur(gray, 3)

    if mode == "adaptive" or (mode == "auto" and spread > 40):
        return cv2.adaptiveThreshold(gray, 255,
                                     cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                     cv2.THRESH_BINARY, 31, 7), "adaptive"

    thr, mask = cv2.threshold(gray, 0, 255,
                              cv2.THRESH_BINARY + cv2.THRESH_OTSU)
    return mask, f"otsu({thr:.0f})"

That corner-spread check is worth keeping. It is four means and a subtraction, and it tells you whether lighting is even before you commit to a method. Log the returned mode across a batch: if it flips between otsu and adaptive on similar images, your capture setup is the thing to fix.

Practice task

Take one photo of a page of text on a desk with a lamp to one side, so half is bright and half is shadowed.

1. Plot the grayscale histogram. Is it bimodal? Where would you put a threshold by eye?
2. Apply a fixed threshold at 128, then at 100, then at 180. Note what breaks in each.
3. Run Otsu and print the chosen value. Compare the mask against your fixed attempts.
4. Run adaptive thresholding with blockSize at 5, 15, 31, and 71. Describe what changes.
5. Compute the corner-mean spread from the pipeline above. Would the auto rule have picked correctly?
6. Run Canny at (50, 150), then (100, 200), then (20, 60). Count the connected components in each.

Step 4 teaches the most. At blockSize=5 the letters come out hollow, which shows you the failure signature described earlier, and once you have seen it you will recognise it instantly in future.

Summary

Thresholding and edge detection answer different questions, and each fails in a characteristic way.

A fixed threshold works only under fixed lighting. Otsu removes the magic number by searching every cutoff for the one that maximises between-class variance, which you worked out by hand to land at $t = 2$, exactly in the histogram’s valley. Otsu needs two peaks; without them it returns an arbitrary and unstable value. Adaptive thresholding computes a local cutoff per region and was the only method that handled the shadowed form, producing thresholds of 227 and 110 in the two halves from the same line of code.

Sobel gives gradient magnitude and direction, computed here as 339 at 45° on a diagonal edge, and must be stored in a signed float type. Canny adds noise suppression, thinning to one pixel via non-maximum suppression, and hysteresis with two thresholds so that weak-but-connected pixels survive while isolated strong noise does not.

What comes next

You have a binary mask, and it is not clean. Morphology, contours, and shape analysis covers removing specks, closing gaps, separating touching objects, and turning the result into measurements you can trust. After that, feature matching and stitching moves from measuring one image to relating two.