Visual tokens don't arrive in language models fluent — they have to earn their place.
A new paper from arXiv compares two dominant ways of connecting vision to language inside large models: feeding visual tokens as in-context prompts alongside text, versus injecting them directly into intermediate layers of the model. Researchers ran both approaches under identical training conditions across single-image, multi-image, and video benchmarks, then traced what actually happens to those visual signals as they travel through the network. What they found is that visual tokens start as raw, linguistically unstructured representations — "disguised visual context" — and are progressively reshaped, but the reshaping follows a different path depending on which integration architecture is used.
The distinction matters because the two paradigms capture different frequency characteristics of the visual signal. In plain terms: one approach picks up fine-grained detail better; the other encodes broader patterns differently. That gap isn't cosmetic — it determines which visual features the model can actually use and how well visual representations align with the language space, which is what drives downstream task performance.
The finding that cuts against common assumptions is this: attention allocation alone doesn't explain performance differences. Where the model "looks" is less important than the quality of the visual representation at each layer. Researchers chasing benchmark gains by tuning attention weights may be pulling the wrong lever.
Vision-language models have advanced rapidly, but most architectural comparisons focus on outputs rather than internals. This kind of mechanistic comparison — tracking how representations evolve layer by layer — is closer to what the field needs to move beyond empirical guesswork.