A frozen image encoder bolted onto a text-only language model works, but only three of its four benchmark wins survive the control that matters, and the fix for the fourth one worked only when the visual tokens carrying it were left uncompressed.
Qwen3-8B is a text-only language model. It has no eyes. The standard way to give it some is the recipe LLaVA introduced: take an image encoder that already produces good visual features, freeze it, and train a small network to translate its output into vectors that sit in the language model's input stream as if they were word embeddings. That translator is called a projector, and it is almost the whole trick. Ours is a two-layer network of 21.5 million parameters standing between a 428.2-million-parameter vision encoder and an 8.191-billion-parameter language model.
The build works. It beats LLaVA-1.5-7B on all four benchmarks we scored, under one harness with identical questions and scorers. But "it beats the baseline" is the easy part, and on its own it is not evidence that the model is looking at anything. A vision-language model can score well on a visual benchmark by knowing that footballs are usually white and that questions about kitchens usually mention sinks. So the first thing worth showing is not the scores. It is the scores next to what the same model gets when we take the image away.
Four benchmarks, one harness, the same question strings and the same scorer for every model. The control that carries the weight is the row labelled same model, image withheld: the trained model run on the same questions with the image taken away. It formats answers exactly the way the real model does, so anything it scores is coming from language priors and answer shape rather than from seeing. Switch the panel to "gain from the image" and watch which columns survive.
The MMMU result is the interesting one, because MMMU is the benchmark that looks most like reasoning and it is the one where our model's margin over LLaVA-1.5-7B is largest (44.56 against 34.67). Almost all of that margin is Qwen3-8B being a stronger language model than Vicuna, not our projector working well. The image-withheld control reaches 43 on its own.
Two checks say this is a real ceiling and not a scoring artifact. First, the blind control is not failing to answer: it emits a parseable multiple-choice letter on 94.11% of items, the same rate as the model that can see. Second, scoring by comparing option log-probabilities instead of parsing generated text, which removes answer formatting from the picture entirely, still puts the blind control 18.27 points above the 26.2% chance rate against 21.10 points for the model with the image, over 847 items. The image is worth a little on MMMU. It is not worth much.
The encoder is SigLIP2-so400m at 384 pixels, frozen throughout. It turns an image into 729 patch vectors of 1,152 dimensions each; the projector maps those to the language model's 4,096-dimensional input space, where they occupy a 729-token span in the prompt. Training runs in two stages, following LLaVA's recipe. Stage 1 trains only the projector, on 558,128 caption pairs at a learning rate of 1e-3: the language model is untouched and all the projector has to learn is where in the embedding space visual content belongs. Stage 2 unfreezes the language model at 2e-5 and trains on instruction data, 5,800 steps at an effective batch of 128, which is 742,400 samples, a single pass over 95.3% of the 779,289-row set. The vision tower stays frozen the whole time, on 8 GPUs, seed 42.
TextVQA is where the model fell furthest short. It asks questions whose answers are written somewhere in the photograph, like a brand on a bottle or a number on a jersey, so it depends on resolving small text. Our model scored 63.60 against Qwen3-VL-8B's 83.56, a gap of 19.96 points.
The obvious suspect is resolution. A single 384-pixel view of a photograph makes small lettering a few pixels tall. The standard fix is AnyRes tiling: cut the image into tiles, run the frozen encoder over each tile separately, and hand the language model the tile features together with a view of the whole image. Each tile is encoded at full resolution, so the effective pixel budget goes up several times over.
But tiling changes two things at once, and they are worth separating. More tiles mean more pixels, and they also mean more tokens: at four tiles plus a full view, the image span grows from 729 tokens to 3,977. A common way to hold the token cost down is a pixel shuffle, which folds each 2×2 block of neighbouring patch vectors into one wider vector before the projector sees it, cutting the token count roughly fourfold while keeping the same pixels. That is a free lunch if what the model needed was resolution, and a bad trade if what it needed was tokens.
So: two arms, both retrained through both stages from the same starting weights, over identical pixels and identical tile geometry. The only intended difference is whether the tile tokens are compressed. One arm hands over every tile token, up to 3,977. The other folds them 2×2, up to 1,009. Both inherited the baseline's recipe verbatim: its saved training arguments and model config were diffed key by key against each arm's, and the diff turned up no unexpected differences in either. The step budget and effective batch were taken from the baseline rather than re-derived.
In absolute terms the full-token arm reached 70.72 on TextVQA and the compressed arm 64.72, against the untiled baseline's 63.60. The two arms land 6.004 points apart, which is 6.43 standard errors once the two arms' own sampling errors on 5,000 items are combined (0.6435 and 0.6758 points). Same pixels, same frozen encoder, same recipe, same tile grid: what separates them is how many of those tile tokens actually reached the language model. So the deficit was not simply that the model could not see finely enough. It was that the compressed representation could not carry what it saw.
It matters that the compressed arm is a working model and not a broken one, because a broken run would explain a flat result without teaching anything. It is healthy on both checks available. It beats its own image-withheld floor by 55.84 TextVQA points, so it is plainly using the image. And it finished stage 2 at a training loss of 0.8469 against the baseline's 0.8361, a difference of 0.011. It learned the task about as well; it just cannot read as well.
The aggregate hides something. Tiling did not simply lift a rising tide of items: it moved 745 items from wrong to right and 339 from right to wrong, a net of 406 out of 5,000. Below are worked examples of both directions, with the real predictions. Most of the gains look like exactly what more pixels should buy, a word or a number resolved that was previously guessed. Many of the losses look like something else.
Neither this page nor the run behind it explains the 339 regressions. Tile seams are the obvious hypothesis, since cutting an image into tiles can split a word across two of them, but nothing here tests it. And 949 items were failed by both models, so there is a large pool that neither resolution nor token budget touched.
The tempting summary is "token compression loses the reading signal". The evidence does not support that general a claim, because the compressed arm differed from the uncompressed one in two ways rather than one. The pixel shuffle cuts the token count, and it also changes what the projector reads: instead of a 1,152-dimensional patch vector, the projector receives four of them concatenated, 4,608 dimensions wide. That is a different and harder input for the same-sized network to condition well.
Measuring the projector's output over 2,000 images, it is conditioned worse in the compressed arm on every reading we have. Its total variance is about eight times the uncompressed arm's (5,229,438 against 646,006). A single direction holds 97.25% of that variance, against 95.88% in the uncompressed arm. Setting that dominant direction aside, what remains spreads across an effective 7.436 directions in the compressed arm against 19.12 in the uncompressed one, and it takes only 5 directions to account for 99% of the compressed arm's variance against 42 for the uncompressed arm's.
That is suggestive and it is not a mechanism. Two problems. First, dimensionality is correlational: a representation losing directions at the same time as a score drops does not show that the lost directions were carrying the reading signal, which would need an ablation that was not run. Second, the two effective-dimension estimators disagree about this arm. The participation ratio reads the compressed arm as lower-dimensional, while TwoNN, a nearest-neighbour estimator, reads it as higher: 40.89 against 27.13. A calibration on synthetic clouds of known dimension explains part of the discrepancy, in that the participation ratio recovers a true 100 dimensions as 97.61 while TwoNN reads a known 5-dimensional cloud 1.32 times too high and a known 100-dimensional one at only 0.709 of its true size. It does not resolve which reading applies here. So the supported claim is the narrow one: this compressor loses the gain, not that compression must.
The full-token arm is not free. It carries up to 3,977 image tokens where the compressed arm carries 1,009 and the untiled baseline carries 729, which is roughly four times the compressed arm's inference cost on the image span and five and a half times the untiled baseline's. Training showed the same shape: 8.413 samples per second against 23.48 for the compressed arm and 26.09 for the untiled baseline, so stage 2 took 24.5 hours against 7.9 hours for the untiled baseline. Buying +7.126 TextVQA points with four times the visual tokens is a real trade, not a strict improvement, and the useful next question is where between 1× and 4× compression the gain actually breaks.
The full-token arm reaches 70.72 on TextVQA against Qwen3-VL-8B's 83.56, closing 35.69% of the gap and leaving roughly 12.8 points. Resolution and token budget have now both been pushed, so what is left looks more like training data scale than anything about how the image is presented: Qwen3-VL-8B is trained on vastly more visual data than a single pass over 742,400 instruction samples. The 339 items tiling broke are unexplained, with tile seams an untested hypothesis. And the compressor question above is open in the specific sense that a compression sweep, holding the projector's input width fixed, would separate token count from conditioning.