Camera Calibration and Perspective Correction

Every measurement you take from a photo is wrong until you calibrate. The lens bends straight lines, and a pixel has no fixed size in the real world. Calibration fixes both, and it takes about twenty photos of a printed checkerboard.

Prerequisites: Feature matching and homography, and matrix multiplication.

The problem: measuring gaskets with a phone

A small workshop cuts rubber gaskets and needs to check their diameter to within a millimetre. Buying a measuring microscope is expensive. A phone on a stand, pointing down at a lit surface, costs nothing.

Photograph a gasket you know is exactly 80 mm across. Measure it in the image and you get 340 pixels — but move the gasket to the corner of the frame and the same gasket measures 328 pixels. It has not changed size. The lens has bent it.

A 2.9 mm error on an 80 mm part is a 3.6% error, well outside tolerance. That error is not random noise you can average away — it is systematic, it depends on position, and calibration removes it exactly.

The pinhole model

Strip a camera down to its simplest form: a box with a tiny hole. Light from a point in the world travels in a straight line through the hole and lands on the sensor behind it. Similar triangles give the whole model:

$x = f \cdot \frac{X}{Z}, \qquad y = f \cdot \frac{Y}{Z}$

$X, Y, Z$ are the point’s position in the world, $f$ is the distance from the hole to the sensor, and $x, y$ is where it lands.

Try it. A part 100 mm wide sits 2,000 mm from a camera whose focal length is 800 pixels:

$x = 800 \times \frac{100}{2000} = 800 \times 0.05 = 40 \text{ pixels}$

Move the part to 1,000 mm and it doubles to 80 pixels. Move it to 4,000 mm and it halves to 20 pixels.

That curve carries an important warning. Image size depends on the ratio $X/Z$, so a 100 mm part at 2,000 mm and a 200 mm part at 4,000 mm both measure 40 pixels. One camera cannot separate size from distance. The workshop gets away with it only because the camera is on a fixed stand, so $Z$ is known and constant.

The intrinsic matrix

The pinhole equations assume the image origin sits at the optical axis. In a real image, $(0,0)$ is the top-left corner. Two more terms fix that, and the whole thing packs into a 3×3 matrix:

$K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$

$f_x$ and $f_y$ differ slightly because sensor pixels are not perfectly square. $c_x, c_y$ land near the image centre but rarely exactly on it, because the sensor is never mounted perfectly concentric with the lens.

Project a real point. The part’s corner sits at $X = 120$ mm, $Y = -80$ mm, $Z = 1500$ mm:

$u = f_x \frac{X}{Z} + c_x = 1240 \times \frac{120}{1500} + 960 = 1240 \times 0.08 + 960 = 99.2 + 960 = 1059.2$

$v = f_y \frac{Y}{Z} + c_y = 1238 \times \frac{-80}{1500} + 540 = 1238 \times (-0.0533) + 540 = -66.0 + 540 = 474.0$

So that corner appears at pixel $(1059, 474)$.

The intrinsic matrix also tells you the field of view:

$\text{FOV}_x = 2 \arctan\left(\frac{w}{2 f_x}\right) = 2 \arctan\left(\frac{1920}{2480}\right) = 2 \times 37.7° = 75.5°$

A useful sanity check. If calibration returns an FOV of 140° for a normal phone lens, the calibration is wrong.

graph LR
  A["World point<br/>(X, Y, Z)"] -->|"extrinsics R, t"| B["Camera coordinates"]
  B -->|"divide by Z"| C["Normalised<br/>(x, y)"]
  C -->|"distortion k1..k3, p1, p2"| D["Distorted<br/>(x_d, y_d)"]
  D -->|"intrinsics K"| E["Pixel (u, v)"]

Extrinsics ($R$ and $t$) say where the camera is in the world; they change every time the camera moves. Intrinsics ($K$) and distortion describe the camera itself; they stay fixed as long as you do not change the lens or zoom. Calibration recovers all of them, but only the intrinsics and distortion are worth saving.

Distortion, worked out radius by radius

A real lens is not a pinhole. It bends light more at the edges, so straight lines bow. The standard model corrects a point in normalised coordinates by a radial factor:

$x_d = x \left(1 + k_1 r^2 + k_2 r^4 + k_3 r^6\right), \qquad r^2 = x^2 + y^2$

Take a typical phone lens with $k_1 = -0.25$ and $k_2 = 0.08$, and a point at normalised $(0.4, 0.3)$:

$r^2 = 0.4^2 + 0.3^2 = 0.16 + 0.09 = 0.25, \qquad r = 0.5$ $\text{factor} = 1 + (-0.25)(0.25) + (0.08)(0.25^2) = 1 - 0.0625 + 0.005 = 0.9425$ $x_d = 0.4 \times 0.9425 = 0.377, \qquad y_d = 0.3 \times 0.9425 = 0.2828$

In pixels with $f_x = 1240$ and $c_x = 960$, the ideal position is $1240(0.4) + 960 = 1456$, but the actual position is $1240(0.377) + 960 = 1427$. The point is pulled 29 pixels toward the centre.

Now compute the same displacement across the whole radius:

The shape of that curve is the practical lesson. At $r = 0.1$ the error is a third of a pixel and you would never notice. At $r = 0.7$ it is 90 pixels. If you only ever measure things in the middle 20% of the frame, you can skip calibration. For anything else, you cannot.

Look at the corners. In the ideal grid they are filled; under barrel distortion they are empty, because the corner content has been pulled toward the centre. That is exactly the 90-pixel shift from the table above.

Calibrating with a checkerboard

A checkerboard works because its corners are detectable to sub-pixel precision and their real-world spacing is known exactly. Every corner gives two equations, so a 9×6 board provides 54 corners and 108 equations per photo.

import cv2
import numpy as np
import glob

PATTERN = (9, 6)          # inner corners, not squares
SQUARE_MM = 25.0          # measure your printed board, do not assume

# Real-world corner positions, z = 0 because the board is flat
objp = np.zeros((PATTERN[0] * PATTERN[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:PATTERN[0], 0:PATTERN[1]].T.reshape(-1, 2)
objp *= SQUARE_MM

obj_points, img_points = [], []
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

for path in glob.glob("calib/*.jpg"):
    gray = cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2GRAY)
    found, corners = cv2.findChessboardCorners(gray, PATTERN, None)
    if not found:
        print(f"no board in {path}")
        continue
    corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
    obj_points.append(objp)
    img_points.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, gray.shape[::-1], None, None)

print(f"RMS reprojection error: {rms:.3f} px")
print(f"fx={K[0,0]:.1f} fy={K[1,1]:.1f} cx={K[0,2]:.1f} cy={K[1,2]:.1f}")
print(f"distortion: {dist.ravel()}")
np.savez("calibration.npz", K=K, dist=dist)

Note SQUARE_MM = 25.0. Measure the printed board with a ruler. Printers scale to fit page margins, and a board you believe is 25 mm but is actually 24.2 mm makes every distance you compute 3% wrong, with no other symptom.

How to shoot the calibration photos

The tilt requirement is the one people get wrong. Photograph the board flat-on twenty times and the solver cannot tell a long lens far away from a short lens up close — both produce the same image. Tilting breaks that ambiguity, because perspective foreshortening depends on the focal length in a way scale alone does not.

Reading the reprojection error

calibrateCamera returns an RMS error in pixels: on average, how far the model’s predicted corner positions are from the detected ones.

# Find which individual photos are dragging the error up
for i in range(len(obj_points)):
    proj, _ = cv2.projectPoints(obj_points[i], rvecs[i], tvecs[i], K, dist)
    err = cv2.norm(img_points[i], proj, cv2.NORM_L2) / len(proj)
    print(f"image {i}: {err:.3f} px")

If one photo shows 2.5 px while the rest sit at 0.3 px, drop it and recalibrate. It is almost always a blurred frame or a board that flexed.

Undistorting and measuring

data = np.load("calibration.npz")
K, dist = data["K"], data["dist"]

img = cv2.imread("gasket.jpg")
h, w = img.shape[:2]

newK, roi = cv2.getOptimalNewCameraMatrix(K, dist, (w, h), alpha=0)
undistorted = cv2.undistort(img, K, dist, None, newK)

x, y, rw, rh = roi
undistorted = undistorted[y:y+rh, x:x+rw]

alpha controls what happens to the edges. At alpha=0 the output is cropped so every pixel is valid — use this for measurement. At alpha=1 all original pixels are kept and the borders curve, leaving black wedges — use this when you must not lose any field of view.

From pixels to millimetres

Undistortion removes the position-dependent error. Converting to real units needs one known reference in the scene, at the same distance as what you are measuring.

Place a 100 mm calibration bar next to the gasket. It spans 412 pixels:

$\text{scale} = \frac{100 \text{ mm}}{412 \text{ px}} = 0.2427 \text{ mm/px}$

The gasket now measures 340 pixels in the undistorted image:

$340 \times 0.2427 = 82.5 \text{ mm}$

The spread collapsed from 2.8 mm to 0.2 mm. That is the entire value of calibration in one row.

Perspective correction without calibration

There is a shortcut for one common case. If you are photographing a flat object and you know the real-world coordinates of four points on it, you can rectify to a top-down view with a homography — no calibration needed.

# Four corners of a document as they appear in the photo
src = np.float32([[214, 88], [1102, 141], [1183, 856], [147, 782]])

# Where they should be: an A4 page at 2 px per mm
W, H = 420, 594
dst = np.float32([[0, 0], [W, 0], [W, H], [0, H]])

M = cv2.getPerspectiveTransform(src, dst)
flat = cv2.warpPerspective(img, M, (W, H))

Four point pairs give exactly eight equations for the homography’s eight degrees of freedom, so getPerspectiveTransform solves it directly with no RANSAC needed.

Common calibration failures

Practice task

Print a 9×6 checkerboard and tape it flat to a clipboard.

1. Measure one square with a ruler. Write down the real number, not the intended one.
2. Take 20 photos: varied tilt, varied distance, board reaching all four frame corners.
3. Run the calibration. Record the RMS error, $f_x$, $f_y$, $c_x$, $c_y$, and $k_1$.
4. Compute the field of view from $f_x$ and check it looks sensible for your lens.
5. Print the per-image error. Drop anything above 1.0 px and recalibrate. How much did RMS improve?
6. Now recalibrate using only 5 photos, all flat-on. Compare $f_x$ to the good calibration.
7. Photograph a ruler at the frame centre and again at the corner. Measure a 100 mm span in pixels, before and after undistortion.

Step 6 is the eye-opener. The flat-on calibration reports a small reprojection error and looks fine, but its $f_x$ can be off by 20% or more. A low reprojection error does not mean a correct calibration — it only means the model fits the data you gave it, and flat-on data cannot constrain focal length.

Summary

The pinhole model reduces a camera to $x = fX/Z$. A 100 mm part at 2,000 mm with $f = 800$ px lands 40 pixels wide, and because only the ratio $X/Z$ matters, one camera can never separate size from distance.

The intrinsic matrix $K$ holds $f_x, f_y, c_x, c_y$. You projected a point at $(120, -80, 1500)$ mm to pixel $(1059, 474)$ and derived a 75.5° field of view from $f_x = 1240$.

Radial distortion follows $1 + k_1r^2 + k_2r^4 + k_3r^6$. You computed a factor of 0.9425 at $r = 0.5$, a 29-pixel inward shift, and tabulated the displacement rising from 0.31 px at $r=0.1$ to 89.7 px at $r=0.7$. That curve is why the gasket shrank as it moved toward the corner, and why undistortion collapsed the measurement spread from 2.8 mm to 0.2 mm.

Calibrate from about 20 tilted checkerboard photos covering the whole frame. Check the RMS error, but remember it only tells you the model fits your photos — flat-on views produce a small error and a badly wrong focal length.

What comes next

Calibration gives you one camera’s geometry, which removes distortion and enables measurement at a known distance — but still not depth. Epipolar geometry and stereo vision adds a second camera and turns the disparity between two views into an actual distance in millimetres. Calibration is a hard prerequisite there: stereo depth is only as accurate as the intrinsics feeding it.