Where a vision-language model starts putting words to what it sees

When you show a photograph to a vision-language model, the picture does not arrive as a picture. A vision encoder cuts it into a grid of small squares, called patches, and turns each patch into a long list of numbers. A component called the projector rewrites those lists into the form the language model accepts, and from there they travel through the language model next to your words until an answer comes out.

Somewhere on that path, the fact that a patch shows part of a pizza has to become something the language half of the model can work with. The work behind this page asked where, and how literally. It took one patch's activity at a given point inside Gemma 3 27B, pushed it through a fitted linear map into the space the model uses at the very end when it chooses a word, and read off which vocabulary items score highest for that patch. Then it checked whether the patches a given word fires on are the patches a human would have labelled with that thing.

They often do. Across 133 everyday concepts and 27 points through the model, 1,259 of the 3,591 frozen readouts overlap the labelled region enough to count, on 5,000 held-out photographs they were never fitted on. And the ability rises and falls with depth in a specific way: it climbs through the vision encoder, jumps at the projector, peaks in the early language layers, and fades well before the model produces its answer.

What is being measured

The readout works because the model's own last step is simple: at the end, a vector in the model's internal space is compared against every item in its vocabulary, and the best-matching items are the candidates for the next word. If you can move a patch's activation from somewhere in the middle of the model into that final space, you can ask the same question of the patch. The fitted linear map that does the moving is called a transport here; it is estimated once, per site, and then held fixed.

Turning that into a test needs labelled pictures. The evaluation uses COCO Panoptic, a set of photographs in which every pixel is labelled with what it belongs to, and 133 of its concepts: people, pizza, zebra, sea, snow, vase, and so on. For each concept and each of the 27 sites, a search over a fitting set of photographs picks the one vocabulary item and one score threshold whose fired patches best cover that concept's labelled pixels. The search may not prefer the concept's own name; any item in the vocabulary can win, and the winner is whichever scores best. The fitting photographs and the held-out photographs come from different halves of COCO, its training and validation splits.

The winning item and threshold are then frozen and scored on all 5,000 held-out photographs, as overlap: of all the pixels that are either labelled as the concept or fired on by the readout, the share where both agree. Overlap of 1 would be a perfect segmentation; overlap near 0 means the two have almost nothing in common. That gives 133 concepts × 27 sites = 3,591 frozen detectors, each with one honest held-out number.

The ability peaks early in the language model

1,259 of the 3,591 detectors clear the 0.04 bar, covering 119 of the 133 concepts; 14 concepts never get there at any site. Taking each concept's best site, the mean overlap is 0.1414 and the median 0.1169. The strongest single detector is sea at language layer 4, at 0.4406 overlap.

The shape with depth is the point. At vision block 10, 13 detectors clear the bar and the average overlap over all candidates is 0.01889. By the last vision blocks it is 43 detectors at block 24 and 43 at block 26. The projector, which exists only to hand patches to the language model, jumps to 65 detectors and 0.06493 average overlap. The peak is 93 detectors at both language layer 12 and language layer 16, with average overlap 0.10911 at layer 12, roughly six times the vision-block-10 figure. From there it falls away, down to 14 detectors at language layer 57, the deepest layer sampled.

Image content is most word-like not where the vision encoder finishes and not where the model speaks, but a short way into the language model, once the patches have been rewritten into language-model form and mixed for a few layers.

What it looks like on real photographs

Aggregate overlap hides how a readout succeeds or fails, so here are individual held-out photographs with the readout's fired region drawn against the human labels.

The transport earns its keep early, not late

There is a simpler thing to try. At a late layer the model is nearly finished, so you can skip the fitted transport and compare the patch's activation against the vocabulary directly, the same way the model picks its next word. That comparison was run at three late language layers, 49, 53 and 57, with the same search, the same fitting set and the same held-out scoring.

At those three layers, plain decoding is the better method, and clearly so: it has higher overlap in 361 of the 399 concept-and-layer comparisons, by 0.0438 overlap on average. But allowing each method its own best site tells the opposite story, because the transport can use sites plain decoding cannot reach: the transport wins 102 concepts to 31, by 0.0546 overlap on average and 0.0358 at the median, against a best-per-concept mean of 0.0868 for plain decoding.

Those two results are the same fact seen twice. Late in the model, where the residual stream is already close to vocabulary space, the fitted map is an unnecessary detour. Its value is that it reads patches early, where the direct comparison has nothing to say. Pizza makes the contrast concrete: 0.4108 overlap at language layer 12 against 0.1109 for plain decoding at its own best late layer, and zebra is readable at the projector at 0.3834, before the language model has seen it at all.

The words the search picked are not always the concepts' names

Because the search was free to choose any vocabulary item, what it chose is itself a result. Sometimes it picks the obvious word, sometimes an emoji, sometimes an ingredient or a texture rather than the object.

The concept names are COCO's own, including merged classes such as tree-merged. Zebra is read as a stripe, pizza as an ingredient, sea and elephant and banana as emoji. The readout is not looking up a label; it is finding whichever piece of the model's vocabulary happens to point in the same direction as those patches.

The failures are just as informative. These readouts did fit their fitting set, then did not survive the held-out photographs.

Apple is read as Pear, vase as a fragment of antique, traffic light as GFP (a green fluorescent protein), frisbee as a French word for whitish. Each one lost most of its overlap between the fitting set and held-out data, which is what picking a single item and threshold out of a very large vocabulary does when the underlying signal is weak.

What this shows, and what it does not

The finding is a claim about where information sits and how legible it is, not a claim about how Gemma works when it answers you.

• The absolute overlap is modest. A best-per-concept mean of 0.1414 and a median of 0.1169 make this a localisation and readout result, not a segmentation system. The strongest detector anywhere reaches 0.4406; the typical concept is far below that.
• The words are discovered, not assigned. Because the search picks whichever vocabulary item scores best, a winning item need not name the concept, and the match between a concept and its item is not evidence that the model uses that item.
• The headline comparison is asymmetric. The transport was searched over all 27 sites; plain decoding was fitted at only three late layers, and it searched candidate pools that had been discovered through the transport. Its measured advantage at those layers is therefore a lower bound, and the transport's best-over-sites win is partly a difference in how much each method was allowed to search.
• Nothing here is causal. No intervention was run. That a patch's content can be read out as a word does not show that the model reads it, or that this information reaches its answer.
• Some controls were not run. The evaluation has no spatial-shuffle control, no bootstrap intervals on the overlap numbers, and the photographs used to fit the transport were not excluded from the held-out pool, so a small overlap between fitting and evaluation images is possible.
• One model, one label set. Everything here is Gemma 3 27B on COCO Panoptic concepts. Whether the same depth profile appears in other vision-language models is untested.

What survives all of that is the shape of the curve. Every concept in the explorer above is scored on held-out photographs with a detector frozen beforehand, and the 1,259 that clear the bar are concentrated at the projector and the first stretch of the language model. If you want to find where a vision-language model has turned a picture into something word-shaped, that is where to look, and it is not where the model is about to speak.