Multimodal Vision: CLIP, Embeddings, and Retrieval Systems

A clothing retailer has 400,000 product photos and a search box that only matches the words someone typed into a spreadsheet three years ago. Search for “beige linen shirt” and you get nothing, because the product was tagged “sand cotton-blend top”. Every classifier so far needed a fixed list of classes and labelled examples for each. This problem has no fixed list.

Prerequisites: Vision transformers and your first image classifier.

One space for two things

The trick is a single vector space that both images and sentences map into.

graph LR
  A["Product photo"] --> B["Image encoder<br/>ViT-B/32"]
  C["'beige linen shirt'"] --> D["Text encoder<br/>Transformer"]
  B --> E["512-d vector"]
  D --> F["512-d vector"]
  E --> G["Same space<br/>compare with cosine"]
  F --> G

Once both live in one space, search is arithmetic. Encode the query sentence, compare it against every stored image vector, sort.

Cosine similarity, computed

Similarity is the cosine of the angle between two vectors:

$\text{sim}(a, b) = \frac{a \cdot b}{\|a\|\,\|b\|}$

Take a 4-dimensional image embedding and three candidate text embeddings.

$a = [3,\; 4,\; 0,\; 0], \qquad \|a\| = \sqrt{9 + 16} = 5$ $\hat{a} = [0.600,\; 0.800,\; 0.000,\; 0.000]$

Candidate 1 — “a photo of a beige linen shirt”:

$t_1 = [2,\; 3,\; 1,\; 0], \qquad \|t_1\| = \sqrt{4+9+1} = \sqrt{14} = 3.7417$ $\hat{t}_1 = [0.5345,\; 0.8018,\; 0.2673,\; 0.0000]$ $\hat{a} \cdot \hat{t}_1 = 0.600(0.5345) + 0.800(0.8018) = 0.3207 + 0.6414 = \mathbf{0.9621}$

Candidate 2 — “a photo of a leather boot”:

$t_2 = [0,\; 1,\; 4,\; 2], \qquad \|t_2\| = \sqrt{0+1+16+4} = \sqrt{21} = 4.5826$ $\hat{t}_2 = [0.0000,\; 0.2182,\; 0.8729,\; 0.4364]$ $\hat{a} \cdot \hat{t}_2 = 0.800(0.2182) = \mathbf{0.1746}$

Candidate 3 — “a photo of a garment”:

$t_3 = [1,\; 1,\; 1,\; 1], \qquad \|t_3\| = 2, \qquad \hat{t}_3 = [0.5,\;0.5,\;0.5,\;0.5]$ $\hat{a} \cdot \hat{t}_3 = 0.600(0.5) + 0.800(0.5) = \mathbf{0.7000}$

The practical shortcut. If you normalise every vector to unit length once, when you store it, then cosine similarity is exactly the dot product — no division at query time. Every production system does this, and it is the single most useful implementation detail in retrieval.

emb = model.encode_image(images)
emb = emb / emb.norm(dim=-1, keepdim=True)   # do this ONCE, at index time
# later: similarity is just a matmul
scores = query_emb @ index.T

Zero-shot classification

There is no separate mechanism for this. You compare the image against sentences instead of against stored images.

labels = ["shirt", "trousers", "shoes", "bag"]
prompts = [f"a photo of a {c}, a type of clothing." for c in labels]
text_emb = model.encode_text(clip.tokenize(prompts))
text_emb = text_emb / text_emb.norm(dim=-1, keepdim=True)
probs = (100.0 * image_emb @ text_emb.T).softmax(dim=-1)

The 100.0 is the inverse temperature — CLIP learns it during training and it lands near 100.

What temperature does

Take the three similarities above, 0.9621, 0.1746, 0.7000, and softmax them at different temperatures.

At $\tau = 1$: logits are the raw similarities.

$e^{0.9621} = 2.6172, \quad e^{0.1746} = 1.1908, \quad e^{0.7000} = 2.0138, \quad \text{sum} = 5.8218$ $\text{probs} = [0.4496,\; 0.2045,\; 0.3459]$

At $\tau = 0.1$: logits become 9.621, 1.746, 7.000. Subtracting the max for stability gives 0, −7.875, −2.621.

$e^{0} = 1, \quad e^{-7.875} = 0.000381, \quad e^{-2.621} = 0.07270, \quad \text{sum} = 1.07308$ $\text{probs} = [0.9319,\; 0.000355,\; 0.06775]$

Temperature never changes the ordering, so it never changes top-1 accuracy. It changes how confident the numbers look, which matters whenever a threshold controls a decision. This is the same idea as the temperature scaling used for calibration.

Prompt wording matters

The reason is mundane. CLIP was trained on image–caption pairs scraped from the web, and web captions are sentences, not bare nouns. A bare noun is out of distribution for the text encoder. Wrapping it in a sentence puts it back in.

How CLIP is trained

Take a batch of $N$ image–caption pairs. Encode all of them, giving an $N \times N$ similarity matrix. The diagonal entries are the true pairs. Every off-diagonal entry is a wrong pairing, and the loss pushes the diagonal up and everything else down.

$\mathcal{L}_i = -\log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N} \exp(s_{ij}/\tau)}$

Worked, for one row of a batch of 4. Cosine similarities of image 1 against all four captions, with $\tau = 0.1$:

$\text{sum} = 9897.1 + 22.20 + 6.05 + 12.18 = 9937.5$ $p = \frac{9897.1}{9937.5} = 0.99593$ $\mathcal{L}_1 = -\ln(0.99593) = 0.00408$

Nearly zero loss — this pair is already well learned. Now a badly learned row, where the true pair scores 0.35 and a distractor scores 0.28:

$[2.8,\; 3.5,\; 3.0,\; 2.2] \;\rightarrow\; [16.445,\; 33.115,\; 20.086,\; 9.025], \quad \text{sum} = 78.671$ $p = \frac{33.115}{78.671} = 0.42094, \qquad \mathcal{L} = -\ln(0.42094) = 0.8653$

A loss 212 times larger, so the gradient goes overwhelmingly to the pairs the model has not learned.

Row 2 is the weak one — its diagonal, 0.35, barely beats the 0.30 next to it. Rows 1, 3, and 4 have diagonals far above their neighbours and contribute almost nothing to the gradient.

What a real similarity matrix looks like

The matrix above uses round numbers to make the arithmetic readable. A real trained CLIP model produces something that surprises people the first time they see it.

Here is a zero-shot run: seven images, eight candidate captions, cosine similarity for every pair. The correct pairing for each image is on the diagonal.

The diagonal values are around 0.30. Off-diagonal values are around 0.19. Correct pairings win by roughly 0.10, not by 0.6.

This also explains why the temperature parameter is so important. Dividing by $\tau = 0.01$ turns a 0.30-versus-0.19 gap into 30 versus 19, which after softmax is a decisive difference. Without it, the softmax over raw cosines would be nearly uniform and the loss would carry almost no signal.

Why batch size matters so much

Each row has $N - 1$ negatives, so the batch size is the difficulty of the task.

With 255 random negatives, telling a shirt from a boot is easy and the model learns coarse categories. With 32,767, the batch contains other shirts, and the model must learn what distinguishes beige linen from white cotton. That is where the fine-grained ability comes from.

Building the search system

flowchart TD
  subgraph Offline
  A["400,000 product photos"] --> B["Encode with image encoder"]
  B --> C["Normalise to unit length"]
  C --> D["Build ANN index<br/>HNSW / IVF-PQ"]
  end
  subgraph "Per query"
  E["'beige linen shirt'"] --> F["Encode with text encoder"]
  F --> G["Normalise"]
  G --> H["ANN search → top 200"]
  H --> I["Filter: in stock, size, price"]
  I --> J["Rerank by business rules"]
  J --> K["Show top 24"]
  end
  D -.-> H

Why brute force stops working

Store 400,000 vectors of 512 dimensions as float32:

$400{,}000 \times 512 \times 4 = 819{,}200{,}000 \text{ bytes} = 819.2 \text{ MB}$

Every query is a dot product against all of them:

$400{,}000 \times 512 = 204{,}800{,}000 \text{ multiply-adds}$

Precision and recall at $k$ answer different questions and both matter:

$P@10 = \frac{7}{10} = 0.700 \qquad R@10 = \frac{7}{20} = 0.350$

Precision@10 says 7 of the 10 results shown were good, which is what the user experiences. Recall@10 says only a third of the relevant catalogue made it onto page one, which is what the merchandising team cares about.

Practice task

Use open_clip and a few thousand images from any public dataset.

1. Encode 2,000 images. Confirm every embedding has unit norm after normalising.
2. Verify that the dot product of two normalised vectors matches cosine_similarity exactly.
3. Run text queries and inspect the top 10. Note which query wordings work best.
4. Run zero-shot classification with bare labels, then with "a photo of a {label}". Compare accuracy.
5. Build a template ensemble of 5 prompts per class by averaging their embeddings. Measure the gain.
6. Sweep the temperature over 1, 0.5, 0.1, 0.01. Confirm accuracy does not change but confidence does.
7. Build 30 queries with hand-labelled relevant sets. Compute precision@10 and recall@10.
8. Build an HNSW index with FAISS. Measure latency and recall@10 against brute force.

Step 6 is the one that fixes a common misunderstanding. Watching the probabilities move dramatically while accuracy stays fixed makes it clear that temperature is a presentation choice, not a modelling one.

Summary

CLIP puts images and sentences in one 512-dimensional space, so search is a cosine similarity. The correct caption scored 0.9621, a vague-but-true one 0.7000, and a wrong one 0.1746. Normalise once at index time and every query becomes a plain dot product.

Zero-shot classification is the same operation aimed at label sentences. Temperature sharpens the softmax without ever changing the ranking: at τ=1 the top class got 0.450, at τ=0.1 it got 0.932. Wrapping labels in "a photo of a {label}" is worth about 4 points, and a template ensemble another 3.

Training uses InfoNCE over an $N \times N$ similarity matrix. A well-learned row gave loss 0.00408; a poorly learned one gave 0.8653, 212 times more, so the gradient concentrates where the model is weak. Batch size is the number of negatives, which is why CLIP trained at 32,768.

Four hundred thousand 512-d float32 vectors are 819 MB and 205M multiply-adds per query. HNSW is 33× faster at 0.98 recall; IVF-PQ is 31× smaller at 0.91.

What comes next

One note on the other direction. CLIP learns a shared image–text space in order to match them. Run the same idea generatively — condition an image generator on a text embedding — and you get DALL·E and the diffusion models that followed. They are siblings from the same insight: once images and text live in one space, you can move in either direction between them. Retrieval finds an existing image for your sentence; generation makes one.

Retrieval at 1.2 ms per query is fast because it is one matrix operation over precomputed vectors. Live video is a harder problem: frames arrive whether or not you are ready. Real-time computer vision systems covers latency budgets, what happens when the queue fills up, and why average FPS is the wrong thing to measure.