A Short History of Computer Vision: 1545 to Now

Most history-of-a-field articles are lists of names and years, and you forget them the same day. This one is built differently. We are going to take one small task and hand it to each generation of computer vision in turn, and watch what each one could and could not do.

Prerequisites: What is computer vision. You need to be comfortable with the idea that an image is a grid of numbers.

The task we will hand to every era

Here is the photo. A city street. Stalls with awnings, stone buildings, a hill behind, and roughly a dozen people walking.

The task: draw a box around each person.

You did it in under a second without noticing you did anything. Now watch how hard it is to write down.

graph LR
  A["Photo of a street"] --> B["???"]
  B --> C["Boxes around<br/>each person"]
  style B fill:#fee,stroke:#c00

That middle box is the whole history of the field. Let’s fill it in.

Before computers: the image itself had to be explained

Long before anyone tried to make a machine see, people had to work out what an image is.

Gemma Frisius published a drawing of one in 1545, using it to watch a solar eclipse safely. Leonardo da Vinci sketched the geometry in the early sixteenth century: rays from every point outside cross at the hole and land in reverse on the back wall.

That is the entire idea behind every camera you have ever used. A lens only makes it brighter and sharper; the geometry is identical. The pinhole model is still the equation we use today when we calibrate a camera and convert pixels into millimetres.

The important consequence is one you should hold onto for the rest of this series:

A photo is a projection. Three dimensions are flattened into two, and the depth information is destroyed. Not compressed, not hidden — destroyed. Any method that recovers depth is inferring it from clues, not reading it.

Biology got there first, and it gave us a hint

Vision evolved independently many times — octopuses, insects with compound eyes, chameleons, humans. That alone tells you it is useful enough to be worth a lot of biological investment. Roughly half the cerebral cortex in primates is involved in processing visual information.

In the 1990s, brain imaging produced a finding that shaped how people thought about recognition. Kanwisher and colleagues (1997) found a region of the human brain that responds much more strongly to faces than to houses — the fusiform face area. Epstein and Kanwisher (1998) found a different region that responds to scenes and places rather than faces — the parahippocampal place area.

So the brain does not appear to run one general recogniser. It has specialised machinery for particular categories, sitting on top of shared early processing.

That observation pushed early computer vision toward a specific design: build general early stages that everything shares, then specialised recognisers on top. Which is exactly what Marr proposed.

1963: Roberts, and the first complete pipeline

Larry Roberts’ MIT PhD thesis, Machine Perception of Three Dimensional Solids (1963), is usually called the start of computer vision. He photographed simple blocks — a cuboid and a pyramid — and built a program that went all the way from photo to 3D structure in four stages:

Step 4 is the proof. If the program can draw the blocks from an angle the camera never saw, it must have recovered actual 3D structure, not just copied pixels.

Hand it our street photo, though, and it collapses immediately. Roberts’ method assumes objects have flat faces and straight edges. A person has neither.

The 1970s: Marr, and the idea that organised everything

David Marr’s book Vision argued something that sounds obvious now and was not then: vision is computation, and it can be studied as a sequence of well-defined representations rather than as an unanalysable biological talent.

His proposal was a three-stage pipeline.

flowchart LR
  A["Input Image<br/><br/>perceived<br/>intensities"] --> B["Primal Sketch<br/><br/>edges, bars, blobs,<br/>zero crossings,<br/>boundaries"]
  B --> C["2½-D Sketch<br/><br/>local surface orientation,<br/>depth discontinuities<br/>— from where you stand"]
  C --> D["3-D Model<br/><br/>object-centred,<br/>volumetric,<br/>hierarchical"]

Take a basketball on a gym floor:

1. Image — a grid of brightness values, nothing more.
2. Primal sketch — the outline of the ball, the court lines, the edges of things.
3. 2½-D sketch — the ball’s visible surface is curved and facing you; the floor recedes.
4. 3-D model — it is a sphere. Full stop. Independent of where you are standing.

Why it is 2½-D and not 3-D

This is the part worth understanding properly, because the name is doing real work.

At stage 3 you know the depth and orientation of every surface you can see, from where you happen to be standing. You know the front of the ball bulges toward you. You know nothing whatsoever about its back — you have never seen it.

That is more than a flat image (you have depth) and less than a real 3D model (you have no volume). Hence the half.

Stage 4 is where it becomes viewpoint-independent: “sphere, radius 12 cm” is true no matter where you stand.

Marr’s framework was a genuine advance, and it also had a weak point that took twenty years to become obvious: it assumed the early stages could be computed reliably. In practice, edge detection on real photographs is noisy and ambiguous, so the errors compounded down the pipeline. Give the street photo to a Marr-style system and the primal sketch already contains thousands of edges from awnings, cobblestones and shadows, with no way to tell which ones bound a person.

1999: SIFT, and the shift to features that survive

David Lowe’s SIFT was a change of strategy. Instead of trying to build a full description of the scene, find a modest number of distinctive points and describe each one so precisely that you can recognise it again in another photo.

What makes a keypoint good:

• Distinctive. A corner where three surfaces meet is unique. A patch of blank wall is not.
• Repeatable. You must find the same physical point again from a different angle.
• Well-localised. You can pin it to a specific pixel, not a general area.

A point in the middle of a long straight edge fails the third test: slide along the edge and everything looks the same. This is the same aperture problem that limits optical flow.

SIFT earned its two big invariances deliberately:

So if you photograph a building and then photograph it again from further away and rotated, SIFT finds many of the same physical points in both. That is what makes panorama stitching and Structure-from-Motion possible, and it is covered properly in feature matching.

Hand it the street photo and it works beautifully — for the buildings. For the people it does essentially nothing. SIFT recognises this exact object again. It has no concept of person as a category. Two different people share no matching keypoints at all.

2005: HOG, and the move from instances to categories

Dalal and Triggs asked a different question. Not “is this the same object?” but “does this region look like a person?”

Their insight is the one worth taking away from the classical era:

People vary enormously in colour and texture, and hardly at all in the arrangement of their edges.

Different clothes, different skin, different lighting — the brightness values change completely. But there is always a curved edge at the head, diagonals at the shoulders, verticals down the body, and a split at the legs. So throw away brightness and keep gradient direction.

For every pixel, HOG computes two things:

• Gradient magnitude — how sharp the brightness change is
• Gradient orientation — which way it points

Then it discards magnitude detail and builds, for each small cell, a histogram of which directions the edges point. The full method, with the arithmetic worked through, is in HOG, HOF and MBH.

Now hand it our street photo. It works. Slide a window over the image, compute HOG, ask a linear SVM “person or not”, and boxes appear around the pedestrians. This was the first method that genuinely did our task.

It has clear limits, and they follow directly from the assumption:

2009: Structure-from-Motion at internet scale

Meanwhile the geometry line of work had its own breakthrough. Agarwal and colleagues took thousands of unstructured tourist photographs of the Colosseum, scraped from the internet, taken with different cameras from different places in different weather, and reconstructed a 3D point cloud of the building — recovering, at the same time, where every photo had been taken from.

The mechanism is triangulation. Find the same physical point in several photos (SIFT does this), and the rays from each camera through that point must all meet at one place in 3D. Solve for the camera positions and the point positions together. That is covered in 3D vision.

Two things are worth noticing. First, this is Marr’s stage 4 — an actual viewpoint-independent 3D model — achieved by geometry rather than by recognition. Second, it did not need any category knowledge at all. It never knew it was looking at a Colosseum.

The quiet driver: datasets

Between 2004 and 2012 the most consequential work was not an algorithm.

The ImageNet Large Scale Visual Recognition Challenge ran on a 1,000-class, 1.43-million-image subset. Show the model a photo of a child playing a steel drum; it outputs five guesses; it scores if “steel drum” is among them. Guess “giant panda” instead and it takes the hit.

That was the arrangement that produced the following chart.

2012: the year the graph bends

The two grey bars on the left are the classical approach at its best: a dense grid of HOG and LBP descriptors, coded and pooled, then a linear SVM. Years of careful engineering had brought the error from 28.2% to 25.8%.

AlexNet took it to 16.4% in one step.

Then, once everyone switched, it kept falling — past the human reference point of about 5.1% by 2015, down to 2.25% by 2017.

What actually happened in 2012

The important thing is what didn’t happen: nobody invented convolutional networks in 2012. LeCun’s LeNet was recognising digits in 1989. The architecture was over twenty years old.

What changed was the two inputs it had always needed:

After the breakthrough

The same machinery was then pointed at the harder tasks, roughly in order of how much labelling they need:

graph TD
  A["Classification<br/>what is in this image?"] --> B["Detection<br/>where is each object?"]
  B --> C["Semantic segmentation<br/>what class is each pixel?"]
  C --> D["Instance segmentation<br/>which object is each pixel?"]
  A -.-> E["one label<br/>per image"]
  B -.-> F["boxes +<br/>labels"]
  C -.-> G["one mask<br/>per class"]
  D -.-> H["one mask<br/>per object"]

The distinction at the bottom of that chain is worth stating plainly, because it trips people up. In a photo of a dining room with six chairs:

• Semantic segmentation colours all six chairs the same. It knows what is a chair, not which chair.
• Instance segmentation gives each chair its own colour. It separates them as individual objects.

If you need a count, you need instance segmentation. Both are covered in semantic and instance segmentation.

Accuracy kept climbing well past the ILSVRC years, and the architecture family changed once more along the way:

The final point, ViT-H/14, is a vision transformer — no convolutions. Which means the field has now replaced its winning architecture twice.

2021: language arrives

CLIP changed the shape of the problem again. Instead of training on images labelled with one of 1,000 fixed classes, train on hundreds of millions of image–caption pairs from the web, and learn to put an image and its caption near each other in a shared space.

flowchart TD
  subgraph Training
    T1["Caption:<br/>'pepper the aussie pup'"] --> TE["Text encoder"]
    I1["The matching photo"] --> IE["Image encoder"]
    TE --> M["Pull matching pairs together,<br/>push mismatched pairs apart"]
    IE --> M
  end
  subgraph Use
    C["Class names:<br/>plane, car, dog, bird"] --> P["'a photo of a {class}'"]
    P --> TE2["Text encoder"]
    Q["New photo"] --> IE2["Image encoder"]
    TE2 --> S["Highest similarity wins"]
    IE2 --> S
  end

The consequence is that the set of classes is no longer fixed at training time. You define the classes by typing them. A model that has never been trained on your categories can still classify into them, because it has learned the general relationship between pictures and words. That is covered in multimodal vision.

The same period produced generative models running the other direction — DALL·E, published the same day as CLIP, generating images from text descriptions.

And quietly, alongside all of this, low-level vision changed what a phone camera can do. A night scene that an iPhone XS rendered as near-black, a Pixel 3 with Night Sight rendered as a usable photograph — by aligning and combining a burst of frames rather than by having a better sensor. That is computer vision doing the work that optics used to do.

What this history tells you to do today

Here is the practical payoff. Each era’s method is still the correct choice for some problems.

Practice task

This takes about forty minutes and it is worth more than re-reading the article.

1. Find a photo of a street scene with several people in it.
2. Write down, in plain English, the rule you would give someone to find the people — no vision knowledge allowed, just instructions.
3. Now find a counter-example in the same photo where your rule fails. There will be one. Someone sitting, someone partly behind a stall, someone in an unusual colour.
4. Add a rule to fix that case. Find a new counter-example.
5. Repeat until you run out of patience. Count your rules.

Most people stop somewhere between four and eight rules, with the photo still not fully handled. That number is the honest measure of why the field took fifty years, and why “learn the rules from examples” eventually won.

Then, for contrast: take two photos of the same building from different positions and think about which physical points you could identify in both. That is a task where hand-written geometry still wins outright, and always will.

Summary

The history of computer vision is a sequence of assumptions being removed.

Roberts assumed flat faces and straight edges. Marr assumed the early stages could be computed reliably. SIFT assumed you were looking for the same physical object again. HOG assumed a category with a consistent edge layout. CNNs removed the assumption that a human has to specify what the category looks like — and replaced it with a requirement for a great deal of labelled data. CLIP removed the assumption that the class list is fixed in advance.

Each step traded a piece of hand-written knowledge for a piece of learned knowledge, paid for in data. Nothing was thrown away: the geometry from the 1990s runs inside every panorama app, and a threshold on a controlled production line still outperforms a network that costs a hundred times more.

What comes next

The next post, math for CV beginners, covers the small set of mathematical ideas that appear in every method above: image shapes, what a convolution actually computes, and why normalisation matters. Marr’s primal sketch, SIFT’s scale space, HOG’s gradients and every CNN layer are all convolutions, so it is worth doing properly.

If the storage side interests you more, how images and video are stored explains what JPEG does to your pixels before your code ever sees them — which matters more than most people expect.