CV Project Workflow: Dataset, Baseline, Error Iteration

Most computer vision projects do not fail on the model. They fail because nobody wrote down what “working” means, the split leaked, or three months went into tuning a model when the real problem was 31 night-time photos. This post is the loop that avoids all three.

Prerequisites: Vision metrics, because every step here depends on being able to measure honestly.

The project we will run through

A scooter-sharing company has a problem. Riders park scooters badly: blocking pavements, lying on their side, or left on private property. The app asks every rider to photograph the scooter at the end of the trip. The company wants those photos checked automatically.

That is the whole brief. It is not enough to start on, and turning it into something buildable is step one.

graph TD
  A["0. Write the spec<br/>decision, cost, target"] --> B["1. Audit 50 images<br/>by hand"]
  B --> C["2. Split by session<br/>train / val / test"]
  C --> D["3. Baseline<br/>cheapest thing that works"]
  D --> E["4. Measure on val"]
  E --> F["5. Bucket the errors<br/>count each cause"]
  F --> G["6. Fix the biggest bucket"]
  G --> E
  E -->|"target met"| H["7. Test once<br/>then ship"]

The loop between steps 4, 5, and 6 is the project. Everything else happens once.

Step 0: write the spec

Before any code, answer five questions in writing. If you cannot, you do not have a project yet.

That last row is the actual goal, and notice it is two numbers, not one. “High accuracy” would have been useless. “Recall ≥ 0.85 at a review rate ≤ 0.20” is something you can test against.

The cost asymmetry is already telling you something. A miss costs 30 times a false alarm, so this project should be tuned for recall. If someone later proposes optimising F1, you have the numbers to explain why not.

Step 1: look at 50 images by hand

Open fifty photos, one at a time, and write down what you see. Not a script. Your eyes. This takes forty minutes and it changes the project.

Here is what that audit produced:

Two findings change the plan immediately.

Only 8 of 50 photos are the clean case. Any demo built on hand-picked good images will mislead everyone about how hard this is.

Five photos are genuinely ambiguous. A scooter on a slope leans, but it is parked correctly. Until you decide how to label those, two labellers will disagree and your ground truth will be noise. Write the rule down: “tilt is measured relative to the ground plane, not the image frame; a scooter upright on its stand is correct regardless of slope.”

labelling guideline A written document giving the exact rule for every ambiguous case, with example images. Without one, different labellers produce contradictory ground truth and no model can fit it. is the most undervalued artifact in vision projects. Write it during this audit, while the ambiguous cases are in front of you.

Step 2: split the data without leaking

data leakage When information from the validation or test set influences training, making the reported score better than what production will deliver. is the most common reason a model that scored 0.94 in testing scores 0.71 in production.

The obvious split is wrong here:

# WRONG for this project
train, temp = train_test_split(all_photos, test_size=0.3, random_state=42)
val, test = train_test_split(temp, test_size=0.5, random_state=42)

Riders often photograph the same scooter at the same docking spot several times in a week. A random split puts near-identical photos in both train and validation. The model memorises that background and reports a great score for recognising a location, not a parking violation.

Split by something the model cannot memorise:

import pandas as pd
import numpy as np

df = pd.read_csv("photos.csv")     # columns: path, label, location_id, date

locations = df["location_id"].unique()
rng = np.random.default_rng(42)
rng.shuffle(locations)

n = len(locations)
train_loc = set(locations[: int(0.70 * n)])
val_loc   = set(locations[int(0.70 * n) : int(0.85 * n)])
test_loc  = set(locations[int(0.85 * n) :])

train = df[df.location_id.isin(train_loc)]
val   = df[df.location_id.isin(val_loc)]
test  = df[df.location_id.isin(test_loc)]

assert not (set(train.location_id) & set(val.location_id))
assert not (set(train.location_id) & set(test.location_id))
print(len(train), len(val), len(test))

The two assert lines are worth keeping permanently. Leakage is silent, and an assertion is the only thing that makes it loud.

Watch leakage happen

Toggle between the correct and leaky pipeline. The leaky version computes normalization statistics before splitting, so validation information reaches the training data. It is the same class of mistake as splitting randomly across sessions: a small ordering error that inflates every number downstream.

Step 3: build a stupid baseline

Before any model, build the cheapest thing that produces an answer. Three levels, in order:

Level 0 — the constant. Always predict the majority class. If 22% of photos are bad parks, always predicting “fine” gives 78% accuracy and 0 recall. This is the number every future result must beat, and it is one line of code.

Level 1 — one hand-made feature. Bad parks are often scooters lying down, which are wider than they are tall. Find the largest contour, measure its aspect ratio, threshold it. An hour of work using thresholding and contours.

import cv2, numpy as np

def looks_fallen(path, ratio_thresh=1.6):
    img = cv2.imread(path)
    if img is None:
        return None
    img = cv2.resize(img, (640, 480))
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray = cv2.GaussianBlur(gray, (0, 0), 1.5)
    _, mask = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
    cnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    if not cnts:
        return False
    x, y, w, h = cv2.boundingRect(max(cnts, key=cv2.contourArea))
    return (w / h) > ratio_thresh

Level 2 — a small pretrained model. Fine-tune a ResNet-18 for twenty minutes, covered in your first image classifier.

Measured on the validation split:

Nothing is inside the target region yet. But the baselines were cheap, they gave a real number to improve on, and the aspect-ratio rule at 0.41 recall told you something important: a third of bad parks really are just fallen scooters, and simple geometry finds them.

Step 4 and 5: measure, then bucket the errors

The ResNet-18 baseline got recall 0.62 on 200 validation photos containing bad parks. That means 76 misses. The instinct is to train longer or try a bigger model. Do neither yet. Open all 76 and sort them into buckets.

Work out that last column for the night bucket. There are 200 positives and the model currently gets $0.62 \times 200 = 124$ of them. Fixing all 31 night errors gives:

$\text{recall} = \frac{124 + 31}{200} = \frac{155}{200} = 0.775$

Error counts per bucket:

Now the picture is completely different from “the model needs to be better”. Night photos alone are 41% of every failure. And the audit in step 1 already told you 14 of 50 photos are taken at night, which is 28% of all traffic. The model was never given enough night data to learn from.

Two buckets deserve special attention

The 9 wrong labels. These are cases where the model was right and the ground truth was wrong. That is 4.5% of the validation positives, which caps your measurable recall at roughly 0.955 no matter what you build. It also means the training set has similar noise, so the model is being actively taught wrong answers. Fixing labels is usually the cheapest available improvement.

The 2 genuinely unclear cases. These define the ceiling. If a careful human cannot decide, no model will. Take them out of the metric or accept them as permanent loss, but do not spend a week on them.

Choose the fix by gain per unit of effort

The two-hour augmentation change returns ten times more per day than the bigger model, and twenty times more than the two-week detector rebuild. It is also the least intellectually exciting item on the list, which is exactly why teams skip it. Do it first. Augmentation strategy is covered properly in vision data pipelines and augmentation.

Step 6: iterate, and keep a log

Run the loop again after each change. Record every experiment, including the failures, in one table.

The biggest single jump is experiment 4: adding night photos, worth +0.11. That was the top error bucket. The model architecture change in experiment 5 was worth +0.01 and quadrupled inference cost, so it was dropped.

graph TD
  M["Measure on validation"] --> E["Open every error"]
  E --> B["Sort into 4-8 buckets"]
  B --> C["Count each bucket"]
  C --> R["Estimate recall gain<br/>if that bucket were fixed"]
  R --> F["Divide gain by effort"]
  F --> D["Do the highest ratio"]
  D --> L["Log the result,<br/>kept or not"]
  L --> M

Touch the test set exactly once

The validation set was used seven times. It is now partly tuned-to, and its 0.89 is slightly optimistic. That is fine and expected — that is what a validation set is for.

The test set is different. Run it once, at the end, and report that number. If it comes out at 0.84 against a 0.85 target, resist the urge to make one more change and re-run. The moment you do, the test set becomes a second validation set and you no longer have an honest estimate of anything.

What this looks like in production

Shipping is not the end of the loop. It is the loop running with new inputs.

graph TD
  A["Model in production"] --> B["Log every raw score"]
  B --> C["Weekly: plot the score distribution"]
  C -->|"distribution shifted"| D["Something physical changed:<br/>camera, lighting, product, season"]
  C -->|"stable"| E["Sample 100 photos,<br/>label them, measure"]
  D --> F["Add the new condition<br/>to training data"]
  E -->|"below target"| F
  F --> G["Retrain, validate, ship"]
  G --> A

Logging the raw score is the key habit, because it needs no labels. Accuracy needs ground truth you rarely have in production, but the distribution of scores is free. When it shifts, you know something changed before any complaint reaches you.

Practice task

Run the loop on something small enough to finish in a weekend.

1. Collect 150 photos of two categories with your phone, deliberately including hard cases: bad light, partial occlusion, odd angles.
2. Write a one-page labelling guideline covering your three most ambiguous cases.
3. Split by capture session, not randomly. Add the assertion lines.
4. Build all three baselines: constant, one hand-made feature, and a fine-tuned model.
5. Open every validation error and bucket them. You will need four to eight buckets.
6. Compute the recall you would reach if each bucket were fully fixed.
7. Do the highest gain-per-effort fix, re-measure, and log it.
8. Repeat twice more, then run the test set exactly once.

The step people skip is 5. It is also the only one that tells you what to do next, and it takes about twenty minutes.

Summary

The workflow is a loop with a fixed order and one rule: never guess what to fix.

Write the spec first, including the cost of each mistake and a numeric target with two numbers in it. Look at fifty images by hand before writing code, and write the labelling guideline while the ambiguous cases are in front of you. Split by session, location, or date so the model cannot memorise a background, and assert that the splits do not overlap. Build the cheapest baseline that produces an answer, so every later result has something to beat.

Then measure, open every error, bucket them, and count. In the example, one bucket held 41% of all failures and a two-hour augmentation change returned ten times more per day of effort than swapping in a bigger model. Log every experiment, including the ones that failed. Touch the test set once.

What comes next

You have the loop. The next posts fill in the techniques it calls for. Edge detection and thresholding and morphology and contours give you strong classical baselines. Your first image classifier and vision data pipelines and augmentation cover the learned side, including the augmentation that gave the best return in the example above.