Semantic and Instance Segmentation: U-Net to Mask R-CNN

A bounding box around a road crack tells you almost nothing useful. Cracks are thin and irregular, so the box is 90% intact tarmac. To budget for resurfacing you need the actual area, which means a decision per pixel.

Prerequisites: YOLO object detection and CNN architectures.

The problem: measuring road damage

A council photographs its roads from a vehicle-mounted camera and needs to know how many square metres of each road are cracked, to allocate resurfacing budget.

For area, semantic is enough and simpler. For counting individual potholes, or when two merge visually, you need instance.

U-Net

The problem with a plain classification backbone is that it throws away spatial resolution. ResNet’s output is 7×7 for a 224×224 input, and you cannot recover pixel-accurate edges from that.

U-Net’s answer: go down as usual, then come back up, and at each step of the way up, concatenate the matching feature map from the way down.

graph TD
  I["Input 256×256×3"] --> E1["enc1: 256×256×64"]
  E1 --> P1["pool → 128×128"]
  P1 --> E2["enc2: 128×128×128"]
  E2 --> P2["pool → 64×64"]
  P2 --> E3["enc3: 64×64×256"]
  E3 --> P3["pool → 32×32"]
  P3 --> B["bottleneck: 32×32×512"]
  B --> U3["up → 64×64×256"]
  U3 --> C3["concat with enc3<br/>64×64×512"]
  C3 --> D3["dec3: 64×64×256"]
  D3 --> U2["up → 128×128×128"]
  U2 --> C2["concat with enc2<br/>128×128×256"]
  C2 --> D2["dec2: 128×128×128"]
  D2 --> U1["up → 256×256×64"]
  U1 --> C1["concat with enc1<br/>256×256×128"]
  C1 --> D1["dec1: 256×256×64"]
  D1 --> O["1×1 conv → 256×256×1"]
  E3 -.->|"skip"| C3
  E2 -.->|"skip"| C2
  E1 -.->|"skip"| C1

Trace the shapes:

Why the skips matter. The bottleneck at 32×32 knows what is in the image but has lost exactly where to within 8 pixels. enc1 at 256×256 knows precisely where every edge is but has no idea what it belongs to. Concatenating gives the decoder both.

class DoubleConv(nn.Module):
    def __init__(self, cin, cout):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(cin, cout, 3, padding=1, bias=False),
            nn.BatchNorm2d(cout), nn.ReLU(inplace=True),
            nn.Conv2d(cout, cout, 3, padding=1, bias=False),
            nn.BatchNorm2d(cout), nn.ReLU(inplace=True))
    def forward(self, x): return self.block(x)

class UNet(nn.Module):
    def __init__(self, n_classes=1):
        super().__init__()
        self.e1, self.e2, self.e3 = DoubleConv(3,64), DoubleConv(64,128), DoubleConv(128,256)
        self.bott = DoubleConv(256, 512)
        self.u3 = nn.ConvTranspose2d(512, 256, 2, 2)
        self.d3 = DoubleConv(512, 256)          # 256 from up + 256 from skip
        self.u2 = nn.ConvTranspose2d(256, 128, 2, 2)
        self.d2 = DoubleConv(256, 128)
        self.u1 = nn.ConvTranspose2d(128, 64, 2, 2)
        self.d1 = DoubleConv(128, 64)
        self.out = nn.Conv2d(64, n_classes, 1)
        self.pool = nn.MaxPool2d(2)

    def forward(self, x):
        s1 = self.e1(x);           x = self.pool(s1)
        s2 = self.e2(x);           x = self.pool(s2)
        s3 = self.e3(x);           x = self.pool(s3)
        x  = self.bott(x)
        x  = self.d3(torch.cat([self.u3(x), s3], 1))
        x  = self.d2(torch.cat([self.u2(x), s2], 1))
        x  = self.d1(torch.cat([self.u1(x), s1], 1))
        return self.out(x)                      # logits, no sigmoid

The output has no sigmoid because BCEWithLogitsLoss applies it internally in a numerically stable way. Apply sigmoid only at inference.

The imbalance trap

Cracks cover about 1.8% of the pixels in a road photo. In a 256×256 image that is 1,180 crack pixels against 64,356 background pixels.

Never report pixel accuracy on an imbalanced segmentation task. The all-background model scores 98.2%, which sounds excellent and is worth nothing. IoU on the positive class scores it 0, which is correct.

Dice and IoU, computed

Both compare a predicted mask with the ground truth. Take a prediction with TP = 620, FP = 80, FN = 180:

$\text{IoU} = \frac{TP}{TP + FP + FN} = \frac{620}{620 + 80 + 180} = \frac{620}{880} = 0.7045$

$\text{Dice} = \frac{2 \cdot TP}{2 \cdot TP + FP + FN} = \frac{1240}{1240 + 260} = \frac{1240}{1500} = 0.8267$

They are not independent. One converts to the other exactly:

$\text{Dice} = \frac{2 \cdot \text{IoU}}{1 + \text{IoU}} = \frac{2(0.7045)}{1.7045} = \frac{1.409}{1.7045} = 0.8267 \;\checkmark$

Since they rank models identically, the choice is about reporting convention: medical imaging papers use Dice, general vision benchmarks use IoU. State which one you are quoting, because a Dice of 0.82 and an IoU of 0.82 are very different models.

The loss function

Plain BCE on a 1.8%-positive task lets the model reach a low loss by predicting background everywhere. Two fixes, usually combined.

Dice loss optimises the overlap metric directly:

$\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum p_i g_i + \epsilon}{\sum p_i + \sum g_i + \epsilon}$

Notice what is absent: true negatives never appear. A million correctly predicted background pixels contribute nothing, so the loss is driven entirely by the foreground.

class DiceLoss(nn.Module):
    def __init__(self, eps=1.0):
        super().__init__(); self.eps = eps
    def forward(self, logits, target):
        p = torch.sigmoid(logits).flatten(1)
        g = target.flatten(1)
        inter = (p * g).sum(1)
        return 1 - ((2*inter + self.eps) / (p.sum(1) + g.sum(1) + self.eps)).mean()

bce  = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([12.0]))
dice = DiceLoss()
loss = 0.5 * bce(logits, y) + 0.5 * dice(logits, y)

The pos_weight of 12 comes from the imbalance. With 1.8% positive pixels, the exact balancing weight is $98.2/1.8 = 54.6$, but that is usually too aggressive in practice and produces very thick, over-eager masks. Something in the 5–15 range works better.

The combination works because the two losses fail differently. BCE gives stable per-pixel gradients but ignores overlap; Dice optimises overlap but has unstable gradients when the prediction is nearly empty. Together they cover for each other.

Instance segmentation with Mask R-CNN

U-Net cannot count. Two potholes that touch become one connected region in the mask. Mask R-CNN handles that by detecting objects first, then segmenting inside each detection.

graph TD
  A["Image"] --> B["Backbone + FPN"]
  B --> C["Region Proposal Network<br/>~1000 candidate boxes"]
  C --> D["RoIAlign<br/>crop + resample to 14×14"]
  D --> E["Box head<br/>refine coordinates"]
  D --> F["Class head<br/>which class"]
  D --> G["Mask head<br/>28×28 mask per class"]
  E --> H["Per-object: box + class + mask"]
  F --> H
  G --> H

The mask head outputs a fixed 28×28 mask per detected object, which is then resized to fit the predicted box. That is coarse, and it is why Mask R-CNN masks look slightly blobby compared with U-Net’s.

RoIAlign, and the bug it fixed

The original Faster R-CNN used RoIPool, which rounded box coordinates to whole feature-map cells. At stride 16, a box edge at pixel 71 maps to feature coordinate $71/16 = 4.4375$, and RoIPool rounded that to 4.

$4.4375 - 4 = 0.4375 \text{ feature cells} = 0.4375 \times 16 = 7 \text{ pixels of error}$

Seven pixels does not matter for a 300-pixel box’s classification. It matters enormously for a 40-pixel pothole’s mask, where it is nearly 20% of the object. Replacing rounding with bilinear interpolation improved mask AP by about 10 points on small objects, from one change.

SAM is worth knowing about for a practical reason unrelated to deployment: it makes annotation dramatically faster. Click a pothole, get a mask, correct it slightly, move on — often five times faster than drawing polygons by hand.

From mask to square metres

The point of all this. Count the positive pixels and multiply by the real area each pixel covers.

The vehicle camera is calibrated so that at the road surface, 1 pixel spans 0.02 m. So each pixel is:

$0.02 \times 0.02 = 0.0004 \text{ m}^2$

A 30-metre road segment’s mask contains 4,820 crack pixels:

$4{,}820 \times 0.0004 = 1.928 \text{ m}^2$

That last column is the deliverable. It is not a model metric — it is a ranked work order, and it is the only reason the segmentation model exists.

The 0.02 m/px figure comes from camera calibration, and it is only valid on the road plane. Cracks further up the frame have fewer pixels per metre, so the mask must be rectified to a top-down view before counting, or near cracks will be systematically overweighted.

Practice task

Use a public crack, road, or medical segmentation dataset.

1. Compute the positive-pixel fraction. Then compute pixel accuracy for an all-background prediction.
2. Train a U-Net with plain BCE. Record IoU and pixel accuracy. Note the gap in how informative they are.
3. Retrain with pos_weight of 5, 12, 30, and 55. Plot IoU against pos_weight.
4. Retrain with Dice loss alone, then with the 50/50 combination.
5. Verify Dice = 2·IoU/(1+IoU) on your own numbers.
6. Sweep the output threshold over 0.3 to 0.7 and plot IoU. Where is the peak?
7. Remove the skip connections and retrain. Compare boundary sharpness visually.
8. Convert your best mask to real area using a known scale.

Step 7 makes the architecture concrete. Without skips, the output is recognisable but blurry, and boundaries wander by several pixels — you can see exactly what the skip connections were carrying.

Summary

Segmentation predicts a class per pixel. Semantic labels every crack pixel the same; instance separates individual potholes.

U-Net goes down to a 32×32 bottleneck that knows what, then back up, concatenating each encoder map so the decoder also knows where. Channel counts double at each concatenation — 256 + 256, then 128 + 128, then 64 + 64.

On a 1.8%-positive task, an all-background prediction scores 98.2% pixel accuracy and IoU of 0. Report IoU or Dice on the foreground, never pixel accuracy. With TP = 620, FP = 80, FN = 180, IoU is 0.7045 and Dice is 0.8267, and $2 \times 0.7045 / 1.7045$ confirms the relationship exactly.

Dice loss ignores true negatives, which is precisely why it handles imbalance. Combined 50/50 with a weighted BCE it reached 0.741 IoU against 0.694 and 0.658 for either alone. Use about a fifth of the exact inverse frequency for pos_weight.

Mask R-CNN adds instance identity, and RoIAlign’s bilinear sampling replaced RoIPool’s rounding — 7 pixels of error at stride 16, which was crippling for small masks.

What comes next

Every architecture so far has been convolutional, built on the assumption that nearby pixels matter most. Vision transformers drop that assumption entirely: split the image into patches and let every patch attend to every other from the first layer. That is a different set of trade-offs, and importantly a different data requirement.