What is Computer Vision? From Pixels to Decisions

Computer vision is the work of turning an image into a decision. A camera gives you a grid of numbers. Something useful has to come out the other end: a count, a label, a location, a measurement, or an alarm.

Prerequisites: none. If you want the full map of this series first, read the Computer Vision roadmap.

A real problem to start with

A small bottling line fills 40 bottles a minute. A worker watches the line and pulls any bottle whose cap is missing. After six hours the worker gets tired and starts missing bottles.

The factory installs a camera above the line. Every time a bottle passes, the camera takes one photo. Now the question is: can a program look at that photo and answer “cap present” or “cap missing”?

That question is computer vision. Nothing about it requires deep learning yet. Let’s look at what the camera actually hands you.

An image is a table of numbers

The camera does not give you a picture. It gives you a rectangle of measurements. Each measurement is one pixel A single light measurement in an image. In an 8-bit grayscale image, 0 means black and 255 means white. .

Here is a tiny 8×8 grayscale crop from the top of a bottle. The cap is dark metal, the bottle glass behind it is bright.

Read that grid the way the computer does. The numbers near 230 are bright glass. The numbers near 35 are the dark cap. The dark values form a blob roughly in the middle. That blob is the cap, as far as the program is concerned.

Now the same crop from a bottle with no cap:

Every value is above 220. There is no dark blob. That difference is something you can compute.

Solve it by hand

Pick a cutoff. Any pixel below 128 is “dark”. Count the dark pixels in each grid.

For the cap-present grid, go row by row:

Total dark pixels: $0 + 4 + 6 + 6 + 6 + 6 + 4 + 0 = 32$.

The grid has $8 \times 8 = 64$ pixels, so the dark fraction is:

$\text{dark fraction} = \frac{32}{64} = 0.50$

For the cap-missing grid, no value is below 128, so the count is 0 and the dark fraction is $0/64 = 0.00$.

Now write a rule:

$\text{decision} = \begin{cases} \text{cap present} & \text{if dark fraction} > 0.15 \\ \text{cap missing} & \text{otherwise} \end{cases}$

That is a complete, working computer vision system. It has an input (pixels), a feature (dark fraction), a threshold (0.15), and a decision. Every model you will ever build has those same four parts. Deep learning replaces the hand-picked feature and threshold with learned ones. It does not change the shape of the problem.

The decision rule as a pipeline:

graph LR
  A["Camera photo<br/>8x8 grid of numbers"] --> B["Feature:<br/>count pixels below 128"]
  B --> C["Divide by total pixels<br/>= dark fraction"]
  C --> D{"dark fraction<br/>greater than 0.15?"}
  D -->|"yes"| E["Cap present"]
  D -->|"no"| F["Cap missing<br/>reject bottle"]

The same thing in code

import numpy as np

cap_present = np.array([
    [230, 232, 231, 229, 228, 230, 233, 231],
    [231, 229,  40,  38,  41,  39, 230, 232],
    [230,  41,  35,  33,  36,  34,  40, 229],
    [228,  39,  33,  30,  31,  33,  38, 230],
    [229,  40,  34,  31,  32,  34,  39, 231],
    [230,  42,  36,  34,  35,  37,  41, 228],
    [232, 230,  43,  40,  42,  41, 229, 230],
    [231, 233, 230, 229, 231, 230, 232, 233],
], dtype=np.uint8)

def dark_fraction(img, cutoff=128):
    dark_mask = img < cutoff        # True/False for every pixel
    return dark_mask.sum() / img.size

print(dark_fraction(cap_present))   # 0.5

img < cutoff builds a grid of True/False the same shape as the image. .sum() counts the True values because Python treats True as 1. .size is 64. Three lines replaced the whole hand count.

Adding the decision is one more line:

def has_cap(img, cutoff=128, min_fraction=0.15):
    return dark_fraction(img, cutoff) > min_fraction

print(has_cap(cap_present))    # True
print(has_cap(cap_missing))    # False

What the numbers look like across a real batch

One image tells you nothing about whether the threshold is safe. Run the feature over a batch and look at the spread. Here are dark fractions measured on twelve bottles from an actual shift.

Dark fraction per bottle, with the 0.15 cutoff drawn in:

Two things stand out. First, the two groups are well separated, so the feature works. Second, bottle B-10 sits at 0.12, much closer to the line than the other uncapped bottles. If lighting drifts a little, B-10 crosses over and gets misclassified.

That gap between 0.12 and 0.39 is your safety margin. A system with a wide margin survives real conditions. A system with a narrow margin fails the week after you install it. Measuring that margin is more useful than any single accuracy number, and it is the idea behind everything in how to evaluate vision models.

The four questions vision answers

The bottle example asked “is the cap there?” That is one of four question types. Nearly every vision product is built from these.

graph TD
  IMG["One image"] --> C["Classification<br/>What is in it?<br/>output: one label"]
  IMG --> D["Detection<br/>Where is each thing?<br/>output: boxes + labels"]
  IMG --> S["Segmentation<br/>Which pixels belong to it?<br/>output: a mask"]
  IMG --> M["Measurement<br/>How big / far / fast?<br/>output: numbers"]

Those four cover most products. But “what is in this image?” is not one question — it is a ladder of increasingly specific questions, and knowing which rung you are on saves a great deal of wasted effort.

The ladder is worth internalising because effort grows steeply as you climb it. Detection needs boxes. Instance identity needs consistent labelling of the same object across images, which is far more expensive. Activity needs video and temporal labels. Scene understanding needs relationships between objects, which almost nobody annotates.

So the question to ask on any project is not “how good can we get?” but “which rung is the lowest one that answers the business question?” A shop counting customers needs detection. A shop that wants to know how many unique customers visited needs instance identity, which is a different and much harder system.

Choosing the task is the most important decision in a vision project, and people get it wrong constantly. If you only need a count, do not build a segmentation model. If you need a physical measurement in millimetres, no amount of classification accuracy will give it to you: you need camera calibration.

Why vision is harder than the example suggests

The 8×8 crop worked because the lighting was fixed and the bottle was centred. Real cameras do not give you that. The same physical cap produces very different numbers depending on conditions.

Here is the dark fraction for one identical capped bottle photographed under six different conditions:

The same object under changing conditions:

The green bars are handled correctly. The red bars are the same cap, misread as missing. Nothing about the cap changed. Only the light and position changed.

This is the whole difficulty of computer vision in one chart. A human never confuses a wet cap with a missing cap, because a human knows what a cap is. The program only knows “count of pixels below 128”.

The standard sources of trouble have names, and naming them helps because each has a known fix:

Two fixes exist, and the table splits them deliberately. Control the scene, so the variation never happens. Or learn from examples, so the model sees enough variation to handle it. Real systems use both, and the first one is far cheaper than people expect.

Notice the last row. Deformation has no scene-control fix, which is exactly why non-rigid objects — people, animals, cloth, food — resisted classical methods for decades and were the first place learned models clearly won.

Vision is an ill-posed inverse problem

There is a deeper reason none of this is easy, and it is worth stating plainly because it explains why perfect vision is not merely a matter of better engineering.

A camera takes a three-dimensional world and projects it onto a two-dimensional grid. That projection throws away a dimension. Vision is the attempt to run that projection backwards.

A small object close to the camera and a large object far away produce identical pixels. A photograph of a road and a photograph of a photograph of a road produce similar pixels. Shadow and dark paint are indistinguishable from a single view.

Every vision system resolves this by adding assumptions — that surfaces are mostly smooth, that objects are mostly rigid, that light mostly comes from above, that things which look like people are people. Classical methods state these assumptions in code. Learned models absorb them from training data. Neither escapes the need for them, and every failure you will debug is ultimately one of those assumptions not holding.

Classical methods or a learned model?

Once you know the variation you face, the method choice usually decides itself.

The honest advice: build the classical version first, even if you are certain you will need a model. It takes an afternoon, it gives you a number to beat, and it forces you to look at your images closely. Half the time it is good enough. The other half, it tells you exactly which cases the model needs to handle. That baseline habit is the backbone of the CV project workflow.

Where computer vision sits

Two bits of orientation that save confusion later.

Computer vision is not one field. It borrows from several, and which one a given technique comes from tells you a lot about how it thinks.

graph TD
  CV["Computer<br/>Vision"]
  OPT["Optics and physics<br/>how light reaches the sensor"] --- CV
  NEU["Neuroscience<br/>how biological vision works"] --- CV
  COG["Cognitive science<br/>perception and attention"] --- CV
  GFX["Computer graphics<br/>the inverse problem"] --- CV
  ML["Machine learning<br/>learning from examples"] --- CV
  ROB["Robotics<br/>acting on what is seen"] --- CV
  IR["Information retrieval<br/>searching visual content"] --- CV
  style CV fill:#eef,stroke:#33f

Graphics deserves a special note. Graphics takes a scene description and produces an image. Vision takes an image and tries to recover the scene description. They are literally inverse problems, which is why graphics ideas — rendering, projection models, materials — keep turning up in vision papers.

Vision is not a subset of deep learning. The nesting runs the other way round:

graph TD
  AI["Artificial Intelligence<br/>systems that act intelligently"]
  AI --> ML2["Machine Learning<br/>systems that improve from data"]
  ML2 --> DL["Deep Learning<br/>many-layered neural networks"]
  DL --> CNN["CNNs and Transformers<br/>architectures for images"]
  CV2["Computer Vision<br/>a problem domain, not a method"]
  CV2 -.->|"uses"| CNN
  CV2 -.->|"also uses"| GEO["Geometry, optics,<br/>signal processing"]
  style CV2 fill:#efe,stroke:#0a0

Computer vision is a problem domain. Deep learning is one family of tools it uses, and a very effective one, but stereo geometry, camera calibration, and frequency-domain filtering are not machine learning at all and remain the right answer for large classes of problem. Treating “computer vision” and “train a CNN” as synonyms is the single most common way people over-engineer a project that a threshold would have solved.

For how the field arrived at this arrangement — and why each earlier approach was abandoned — see the history of computer vision.

The shape of a full vision system

A deployed system is more than a model. The bottle line looks like this end to end.

graph TD
  A["Camera + lens + lighting"] --> B["Capture and trigger<br/>one frame per bottle"]
  B --> C["Preprocess<br/>crop, resize, normalize"]
  C --> D["Feature or model<br/>produces a score"]
  D --> E["Decision rule<br/>threshold on the score"]
  E --> F["Action<br/>reject arm, log, alert"]
  F --> G["Monitoring<br/>reject rate over time"]
  G -->|"rate drifts"| C

Most production failures happen outside the box marked “model”:

• The lens loses focus after a vibration, and every image goes soft.
• Someone repositions the light, and every brightness value shifts.
• The factory switches to a darker conveyor belt, and the background stops being bright.
• A new cap colour arrives that nobody photographed.

The monitoring loop at the bottom is what catches these. If your reject rate jumps from 2% to 19% overnight, the algorithm almost certainly did not change. Something in front of the lens did.

Common mistakes when starting out

Practice task

Do this before moving on. It takes about thirty minutes and it makes everything after it easier.

1. Take twenty photos of one object on your desk with your phone. Ten with the object present, ten without it. Vary the lighting a little.
2. Load each one in Python and convert to grayscale:

import cv2
import numpy as np

img = cv2.imread("photo_01.jpg", cv2.IMREAD_GRAYSCALE)
print(img.shape, img.dtype, img.min(), img.max())

3. Compute the dark fraction for all twenty and print them next to the true answer.
4. Pick a threshold that separates the two groups. Write down the smallest gap between a “present” value and an “absent” value.
5. Now take five more photos in worse light and see whether your threshold survives.

Step 5 is the point of the exercise. The number you get there is the honest one.

Summary

An image is a grid of numbers, and computer vision is the process of reducing that grid to a decision. You saw the whole loop on real data: a feature (dark fraction), a threshold (0.15), a decision (cap or no cap), and a check on held-out images.

Every vision problem is a version of one of four questions: what is this, where is it, which pixels are it, or how much is it — and “what is this” is itself a ladder from detection through category, instance, attributes, activity and scene. Vision is hard because lighting, viewpoint, scale, and occlusion change the numbers without changing the object, and more fundamentally because projecting a 3D world onto a 2D grid throws away information that no algorithm can recover without assumptions. You handle that either by controlling the scene or by learning from enough examples, and good systems do both.

What comes next

You now have the mental model. The history of computer vision is the natural companion to this post — it walks one problem through every era of the field and shows exactly why each approach was replaced, which makes the rest of the series read as a sequence of answers rather than a list of techniques.

After that, Math for CV beginners gives you the small set of math ideas that appear in every method: how image shapes work, what a convolution actually computes, and why normalization matters. Then image fundamentals covers colour spaces, histograms, and noise removal, and OpenCV setup gets you running code.