CV Explainability: Grad-CAM, Failures, Bias Checks

A hospital’s pneumonia classifier scored 96% on the held-out test set. It was deployed, and accuracy collapsed to 71%. The Grad-CAM maps showed why in about ten seconds: the model was looking at the corner of the image, where the portable scanner burned in a small marker. Portable scanners get wheeled to patients too sick to walk, so that marker predicted pneumonia beautifully — until the model met a different hospital.

Prerequisites: Vision transformers and computer vision metrics.

The setup

A binary chest X-ray classifier: pneumonia or normal. Trained on 12,000 images from one hospital, tested on a held-out split from the same hospital.

That fourth row is the diagnostic. Blacking out a 40-pixel corner cost 14 points of accuracy on the original test set. No genuine medical signal lives in that corner.

Shortcuts are not bugs in training. Gradient descent did exactly what you asked: it minimised loss. The scanner marker was the easiest available route.

Grad-CAM

Grad-CAM answers: which spatial regions of the last conv layer increased the score for this class?

The recipe has three steps.

Step 1. Get the gradient of the class score $y^c$ with respect to each feature map $A^k$.

Step 2. Average each gradient map to a single weight:

$\alpha_k^c = \frac{1}{Z}\sum_i \sum_j \frac{\partial y^c}{\partial A^k_{ij}}$

Step 3. Weighted sum of the feature maps, clipped at zero:

$L^c = \text{ReLU}\!\left(\sum_k \alpha_k^c A^k\right)$

Worked, with four feature maps

Take a toy 2×2 spatial size and four channels. Here are the activations:

And here are the gradients of the pneumonia score with respect to each:

Channel 3 matters most ($\alpha_3 = 0.6$). Channel 2 has a negative weight, meaning it pushes the score down. Channel 4 is nearly irrelevant.

Now the weighted sum, cell by cell:

$L_{00} = 0.2(2) + (-0.15)(9) + 0.6(1) + 0.025(4)$ $= 0.4 - 1.35 + 0.6 + 0.1 = -0.25 \;\xrightarrow{\text{ReLU}}\; 0$

$L_{01} = 0.2(8) + (-0.15)(1) + 0.6(7) + 0.025(4)$ $= 1.6 - 0.15 + 4.2 + 0.1 = 5.75$

$L_{10} = 0.2(1) + (-0.15)(2) + 0.6(0) + 0.025(4) = 0.2 - 0.3 + 0 + 0.1 = 0.00$

$L_{11} = 0.2(1) + (-0.15)(2) + 0.6(2) + 0.025(4) = 0.2 - 0.3 + 1.2 + 0.1 = 1.20$

Top-right is the hot region. That 2×2 map gets bilinearly upsampled to the input size and overlaid.

Why the ReLU. Without it, the negative cell would show as a strong signal in the opposite direction, which is not what you asked. Grad-CAM’s question is “what supports this class”, so evidence against it is discarded.

feats, grads = {}, {}
layer = model.layer4[-1]
layer.register_forward_hook(lambda m,i,o: feats.setdefault('a', o))
layer.register_full_backward_hook(lambda m,gi,go: grads.setdefault('g', go[0]))

logits = model(x)
model.zero_grad()
logits[0, target_class].backward()

alpha = grads['g'].mean(dim=(2,3), keepdim=True)   # step 2
cam = torch.relu((alpha * feats['a']).sum(1))      # step 3
cam = cam / (cam.amax() + 1e-8)
cam = F.interpolate(cam[None], size=x.shape[-2:], mode='bilinear')[0,0]

Two details that trip people up. Hook layer4[-1], the last conv block — earlier layers have finer spatial resolution but their features are edges and textures, not concepts, so the maps look pretty and mean little. And backprop from the logit, not the softmax probability, because the softmax mixes in gradients from the other classes.

What Grad-CAM cannot tell you

Grad-CAM’s spatial resolution at layer4 of a ResNet is 7×7 for a 224×224 input. Each cell covers a 32×32 region. That is enough to say “the corner” and nowhere near enough to say “this specific opacity in the lower left lobe”.

The blackout test is the one that settles arguments. A heatmap is suggestive; re-running the full evaluation with a region masked out gives you a number. Fourteen points of accuracy is not a matter of interpretation.

Slicing metrics

One aggregate number is the enemy. Here is the same 96.2% model, sliced:

The portable-scanner slice at 98.7% is the tell. Portable scans are technically harder — worse positioning, more motion blur, lower dose. A model doing real medicine should score worse there, not better. Scoring better means it found something in those images other than the anatomy.

Failure analysis

Take 60 to 100 errors and read them. Not a sample of the whole set — the errors.

flowchart TD
  A["Collect 60-100 errors"] --> B["Look at each one"]
  B --> C{"What went wrong?"}
  C -->|"label is wrong"| D["Annotation error"]
  C -->|"image is unusable"| E["Data quality"]
  C -->|"never seen this case"| F["Coverage gap"]
  C -->|"looks learnable, model missed it"| G["Model capacity"]
  C -->|"model looked elsewhere"| H["Shortcut"]
  D --> I["Count each bucket"]
  E --> I
  F --> I
  G --> I
  H --> I
  I --> J["Fix the largest bucket first"]
  J --> K["Re-measure"]
  K --> A

The 76 errors from the second hospital, bucketed:

The ranking that matters is errors-removed per day of work, not errors-removed:

Do the top two first. They cost two days, and the border crop is the third — more expensive but it addresses the failure that actually broke deployment.

Calibration

The model says 0.85. Should you believe it?

Bin the test predictions by confidence and check:

Expected calibration error is the size-weighted average gap:

$\text{ECE} = \sum_b \frac{n_b}{N}\left|\text{acc}_b - \text{conf}_b\right|$

$= \frac{120(0.037) + 180(0.057) + 250(0.092) + 300(0.114) + 150(0.114)}{1000}$

$= \frac{4.44 + 10.26 + 23.00 + 34.20 + 17.10}{1000} = \frac{89.00}{1000} = 0.0890$

An ECE of 0.089 means the confidence is off by about 9 points on average.

Temperature scaling

One parameter fixes most of it. Divide the logits by $T$ before the softmax, and fit $T$ on the validation set.

Take logits $[3.0, 1.0, 0.5]$. At $T = 1$:

$e^{3.0} = 20.086,\quad e^{1.0} = 2.718,\quad e^{0.5} = 1.649,\quad \text{sum} = 24.453$ $\text{probs} = [0.8214,\; 0.1112,\; 0.0674]$

At $T = 1.5$, logits become $[2.000, 0.667, 0.333]$:

$e^{2.000} = 7.389,\quad e^{0.667} = 1.948,\quad e^{0.333} = 1.395,\quad \text{sum} = 10.732$ $\text{probs} = [0.6885,\; 0.1815,\; 0.1300]$

Confidence drops from 0.821 to 0.689. Crucially, the ranking is unchanged, so accuracy, AUC, and mAP are all identical. Only the numbers on the confidence become honest.

class TemperatureScaler(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
        self.log_T = nn.Parameter(torch.zeros(1))   # T = exp(0) = 1
    def forward(self, x):
        return self.model(x) / self.log_T.exp()

# Fit T on the VALIDATION set only, with the model frozen
opt = torch.optim.LBFGS([scaler.log_T], lr=0.01, max_iter=50)

Fitting on the test set would be cheating in a subtle way: you would be reporting a calibration you tuned on the data you are measuring.

Practice task

Use any classifier you have trained.

1. Implement Grad-CAM with hooks. Verify your $\alpha_k$ are the channel means of the gradients.
2. Run it on 20 errors and 20 correct predictions. Look for differences in where the model attends.
3. Pick a region the model attends to that should not matter. Black it out and re-evaluate.
4. Slice accuracy by at least three metadata fields. Flag anything more than 5 points below overall.
5. Bucket 60 errors by cause, then rank fixes by errors-removed per day.
6. Build a reliability table with 5 bins and compute ECE by hand.
7. Fit a temperature on validation. Recompute ECE, accuracy, and AUC. Confirm only ECE moved.
8. Deliberately add a shortcut — a small coloured square in every positive training image — and retrain. Watch Grad-CAM find it.

Step 8 is the fastest way to trust the tool. You know the ground truth about what the model learned, so you can see whether Grad-CAM reports it.

Summary

A 96.2% model failed at 71.4% elsewhere because it read a scanner marker. Blacking out a 40-pixel corner cost 14 points on the original test set, which no genuine signal could explain.

Grad-CAM averages the gradients per channel to get weights — 0.200, −0.150, 0.600, 0.025 — then takes the weighted sum of activations and applies ReLU. Cell (0,0) came out at −0.25 and became 0; cell (0,1) at 5.75 became the hot spot.

Slicing exposed paediatric at 84.4% and underexposed at 79.0%, both invisible in the headline. The portable-scanner slice scoring above average was the giveaway, since those images are harder.

Bucketing 76 errors ranked the fixes: labels and exposure at 18 and 17 errors per day, paediatric data at 0.6.

Calibration showed the model claiming 0.851 while being right 73.7% of the time. ECE was 0.0890. A single fitted temperature of 1.62 dropped it to 0.0143 with accuracy and AUC unchanged.

What comes next

Everything so far has treated an image as a flat grid. 3D vision reconstructs the scene itself — recovering camera positions and 3D structure from a set of ordinary photographs, and estimating where an object sits in space.