Vision Metrics: Accuracy, Precision, Recall, mAP, IoU

A model that is 98% accurate can be completely useless. This post shows exactly how that happens, then works out every metric you need by hand on real numbers, so you know what each one is hiding.

Prerequisites: OpenCV setup and first 10 tasks. The general ideas here overlap with evaluation metrics in machine learning, and this post adds the vision-specific ones: IoU, mAP, and Dice.

The 98% model that gets someone fired

A factory inspects circuit boards. Out of 1,000 boards, 30 have a defect. A model is trained to flag defects.

Here is a model that scores 97% accuracy without doing anything at all:

def predict(board_image):
    return "OK"          # always

It is right on all 970 good boards and wrong on all 30 defective ones. Accuracy is $970/1000 = 0.97$. It catches zero defects. Every faulty board ships.

Now here is a real model’s output on the same 1,000 boards:

This table is where every metric comes from. Read the four cells:

• TP = 18: defects the model caught.
• FN = 12: defects it missed. These ship to customers.
• FP = 7: good boards it wrongly flagged. A person wastes time re-checking them.
• TN = 963: good boards correctly passed.

Now compute every metric by hand

Accuracy — of all predictions, how many were right?

$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{18 + 963}{1000} = \frac{981}{1000} = 0.981$

98.1%. Better than the do-nothing model’s 97%, by 1.1 points. That tiny gap is the problem: accuracy barely notices the difference between a model that catches 18 defects and one that catches none.

Precision — when it says defect, how often is it right?

$\text{Precision} = \frac{TP}{TP + FP} = \frac{18}{18 + 7} = \frac{18}{25} = 0.72$

Of 25 flagged boards, 18 really were defective. 7 people wasted their time.

Recall — of all real defects, how many did it find?

$\text{Recall} = \frac{TP}{TP + FN} = \frac{18}{18 + 12} = \frac{18}{30} = 0.60$

It found 60% of the defects. 12 faulty boards shipped. That number is what the factory manager cares about, and accuracy never showed it.

F1 — the harmonic mean, used when you want one number balancing both:

$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = 2 \times \frac{0.72 \times 0.60}{0.72 + 0.60} = 2 \times \frac{0.432}{1.32} = 0.655$

Note how far F1 (0.655) is from accuracy (0.981). They are describing the same model.

Which of precision and recall matters more

They pull against each other. Catching more defects means flagging more boards, which means more false alarms. The right balance comes from the cost of each mistake, not from the maths.

graph TD
  Q{"Which mistake<br/>costs more?"} -->|"missing a real one"| R["Optimise recall<br/>lower the threshold"]
  Q -->|"a false alarm"| P["Optimise precision<br/>raise the threshold"]
  Q -->|"about equal"| F["Optimise F1<br/>or pick by F-beta"]
  R --> T["Then report precision<br/>so the alarm load is visible"]
  P --> U["Then report recall<br/>so the misses are visible"]
  F --> V["Always report both<br/>plus the class balance"]

When the two costs are unequal but you still want one number, use $F_\beta$, which weights recall $\beta$ times as much as precision:

$F_\beta = (1 + \beta^2) \times \frac{P \times R}{\beta^2 P + R}$

With $\beta = 2$ (recall matters four times as much), using our numbers:

$F_2 = 5 \times \frac{0.72 \times 0.60}{4(0.72) + 0.60} = 5 \times \frac{0.432}{3.48} = 0.621$

$F_2$ (0.621) sits closer to recall (0.60) than F1 (0.655) does, which is exactly the intent.

The threshold moves everything

The model does not output “defect” or “OK”. It outputs a score between 0 and 1. You choose where to cut.

At threshold 0.10 the model catches 29 of 30 defects, but raises 210 false alarms — the inspectors would ignore it within a week. At 0.90 every alarm is genuine, but 25 defects ship. The factory has to decide which failure it can live with. No metric makes that decision for you.

Feel the tradeoff yourself

Drag the threshold and watch all four numbers move together. Notice that precision and recall never rise at the same time. That is not a limitation of any particular model; it is what a threshold does.

IoU: the metric that only exists in vision

Classification has one question: is the label right? Detection has two: is the label right, and is the box in the right place? IoU Intersection over Union. The overlapping area between two boxes divided by the total area they cover together. 1.0 means identical, 0 means no overlap. answers the second.

$\text{IoU} = \frac{\text{area of overlap}}{\text{area of union}}$

Work one out completely

The model predicts a box at $(x_1, y_1, x_2, y_2) = (50, 40, 150, 140)$. The ground truth box is $(80, 60, 180, 170)$.

Step 1: find the overlapping rectangle. Take the larger of the two left edges and the smaller of the two right edges:

$x_1^{\cap} = \max(50, 80) = 80, \qquad x_2^{\cap} = \min(150, 180) = 150$ $y_1^{\cap} = \max(40, 60) = 60, \qquad y_2^{\cap} = \min(140, 170) = 140$

Step 2: overlap area.

$w = 150 - 80 = 70, \qquad h = 140 - 60 = 80$ $\text{overlap} = 70 \times 80 = 5{,}600$

Step 3: each box’s own area.

$A_{pred} = (150 - 50)(140 - 40) = 100 \times 100 = 10{,}000$ $A_{gt} = (180 - 80)(170 - 60) = 100 \times 110 = 11{,}000$

Step 4: union. Add the two areas and subtract the overlap once, because otherwise it is counted twice:

$\text{union} = 10{,}000 + 11{,}000 - 5{,}600 = 15{,}400$

Step 5: IoU.

$\text{IoU} = \frac{5{,}600}{15{,}400} = 0.364$

0.364 is below the usual cutoff of 0.5, so this detection counts as a false positive even though it found the right object with the right label. It was simply not placed accurately enough.

def iou(box_a, box_b):
    ax1, ay1, ax2, ay2 = box_a
    bx1, by1, bx2, by2 = box_b

    ix1, iy1 = max(ax1, bx1), max(ay1, by1)
    ix2, iy2 = min(ax2, bx2), min(ay2, by2)

    iw, ih = max(0, ix2 - ix1), max(0, iy2 - iy1)
    inter = iw * ih
    if inter == 0:
        return 0.0

    area_a = (ax2 - ax1) * (ay2 - ay1)
    area_b = (bx2 - bx1) * (by2 - by1)
    return inter / (area_a + area_b - inter)

print(iou((50, 40, 150, 140), (80, 60, 180, 170)))   # 0.36363...

The max(0, ...) on the width and height is essential. Without it, two non-overlapping boxes give negative width times negative height, which is a positive area and a completely wrong IoU.

What IoU values look like

The IoU threshold changes your score dramatically. mAP@0.5 is generous: any roughly correct box passes. mAP@0.5:0.95 averages over ten thresholds from 0.5 to 0.95 in steps of 0.05, and is much stricter because it rewards precise placement. The same model routinely scores 0.62 under the first and 0.41 under the second. Two papers reporting “mAP 0.62” may not be comparable at all if they used different thresholds.

Average Precision, worked out step by step

Average Precision The area under the precision-recall curve for one class. It summarises performance across every possible threshold in a single number. is the standard detection metric, and it is much less mysterious once you compute one.

Suppose a test set contains 5 real objects of one class. The model returns 7 detections. Sort them by confidence, highest first, and mark each as a true or false positive using IoU ≥ 0.5:

Now walk down the list, keeping running totals. At each row, precision is $TP/(TP+FP)$ so far, and recall is $TP/5$.

Recall stops at 0.80, because one of the five objects was never detected at any confidence.

The raw curve zigzags, because each false positive drops precision and each true positive lifts it. AP uses the interpolated version: at every recall level, take the highest precision achieved at that recall or any higher one.

$p_{interp}(0.20) = \max(1.000, 1.000, 0.667, 0.750, 0.600, 0.667, 0.571) = 1.000$ $p_{interp}(0.40) = \max(1.000, 0.667, 0.750, 0.600, 0.667, 0.571) = 1.000$ $p_{interp}(0.60) = \max(0.750, 0.600, 0.667, 0.571) = 0.750$ $p_{interp}(0.80) = \max(0.667, 0.571) = 0.667$

Then AP is the area under that step function. Each recall step is $\Delta R = 0.20$ wide:

$AP = 0.20(1.000) + 0.20(1.000) + 0.20(0.750) + 0.20(0.667)$ $AP = 0.20 \times 3.417 = \mathbf{0.683}$

mAP is just AP averaged over all classes:

$\text{mAP} = \frac{1}{C} \sum_{c=1}^{C} AP_c$

$\text{mAP} = \frac{0.81 + 0.76 + 0.54 + 0.38}{4} = \frac{2.49}{4} = 0.62$

That single 0.62 hides the real story: traffic lights are at 0.38 while people are at 0.81. If your product is about traffic lights, mAP told you nothing useful. Always look at the per-class table, not just the mean. Notice too that the weakest classes have the fewest examples, which points straight at what to collect next.

Segmentation: IoU and Dice

Segmentation predicts a class per pixel, so a bounding box is not enough. The same IoU idea applies, counting pixels instead of area.

Suppose for one image: the predicted mask has 4,200 pixels, the ground truth mask has 3,800, and 3,100 pixels are in both.

IoU (also called the Jaccard index):

$\text{union} = 4{,}200 + 3{,}800 - 3{,}100 = 4{,}900$ $\text{IoU} = \frac{3{,}100}{4{,}900} = 0.633$

Dice coefficient (also called F1 for pixels):

$\text{Dice} = \frac{2 \times \text{overlap}}{|A| + |B|} = \frac{2 \times 3{,}100}{4{,}200 + 3{,}800} = \frac{6{,}200}{8{,}000} = 0.775$

They measure the same overlap and are linked exactly:

$\text{Dice} = \frac{2 \times \text{IoU}}{1 + \text{IoU}} = \frac{2(0.633)}{1.633} = 0.775 \;\checkmark$

Dice always looks better. That is why medical imaging papers usually report Dice and object-detection papers usually report IoU. Neither is wrong, but you cannot compare a Dice number against an IoU number.

graph TD
  A["Segmentation output"] --> B["Per-image IoU or Dice"]
  B --> C["Average over images<br/>= dataset score"]
  B --> D["Average over classes<br/>= mIoU"]
  C --> E["Beware: large objects<br/>dominate the average"]
  D --> F["Beware: rare classes<br/>get equal weight"]

The two averaging choices give different numbers and answer different questions. Averaging over images weights big objects heavily, because they contribute more pixels. Averaging per class first (mIoU) gives a rare class the same weight as a common one. Say which one you used.

Choosing a metric

A metric that beats all of these

For a working product, translate the confusion matrix into money or time. Using the board example with a false negative costing $200 in warranty claims and a false positive costing$5 of inspector time:

$\text{cost} = 12 \times \$200 + 7 \times \$5 = \$2{,}400 + \$35 = \$2{,}435$

Now compare against the threshold-0.30 row from earlier (FN = 4, FP = 74):

$\text{cost} = 4 \times \$200 + 74 \times \$5 = \$800 + \$370 = \$1{,}170$

The lower threshold has worse precision, worse F1, and less than half the cost. F1 said threshold 0.5 was better. The business says otherwise, and the business is right.

def expected_cost(tp, fp, fn, cost_fp=5, cost_fn=200):
    return fp * cost_fp + fn * cost_fn

for thr, tp, fp, fn in [(0.1,29,210,1), (0.3,26,74,4), (0.5,18,7,12),
                        (0.7,12,2,18), (0.9,5,0,25)]:
    print(f"thr={thr}  cost=${expected_cost(tp, fp, fn):,}")

Run it and you will find the minimum sits at 0.30, not at the F1 optimum. Whenever you can put a number on each mistake, do it. It ends most arguments about which threshold to use.

Practice task

Take the coin counter you built in the OpenCV tasks post and evaluate it properly.

1. Label 30 photos by hand with the true coin count.
2. Run your counter and record predicted counts.
3. Build a confusion matrix at the object level: a coin found is a TP, a coin missed is an FN, an extra blob is an FP.
4. Compute precision, recall, and F1 by hand for at least one image before trusting any library.
5. Sweep your area threshold from 100 to 2,000 in steps of 100 and plot precision and recall against it.
6. Assign a cost to each error type and find the area threshold that minimises total cost.

Step 5 is the one that teaches the most. You will see precision and recall cross somewhere, and that crossing point is almost never where you would have guessed.

Summary

Accuracy is the metric to distrust first. On a problem with 3% positives it rates a do-nothing model at 97%, and it moved by only 1.1 points between a model that catches nothing and one that catches 18 of 30 defects.

Precision and recall separate the two ways of being wrong, and the threshold moves you along the curve between them. IoU adds the vision-specific question of whether the box or mask is in the right place, and you computed one at 0.364 that looked correct but failed the standard 0.5 gate. Average Precision folds the whole precision-recall curve into one number, and you worked one out to 0.683 from a ranked list of seven detections. mAP averages that across classes and hides per-class disasters, so always read the breakdown. Dice and IoU measure the same thing and are related by $\text{Dice} = 2\text{IoU}/(1+\text{IoU})$, with Dice always the friendlier figure.

When you can price the mistakes, expected cost beats all of them.

What comes next

The CV project workflow puts these metrics to work inside a loop: build a baseline, measure it, look at the failures, fix the biggest cause, and measure again. That loop is what actually turns a 0.60 recall into a 0.90 recall, and it starts with the numbers from this post.