A new benchmark suggests AI vision models can't reliably count things because of how they're built, not what they're shown.
Researchers built NumerosityVLM, a diagnostic test of 10,800 synthetic images across six controlled conditions that separately vary object size, spatial arrangement, and count while stripping away texture, shape, and color cues. Testing seven vision-language models with no fine-tuning, they found that which architecture a model uses accounts for far more of the performance gap than how the images actually look. Layer-by-layer probing showed a linearly separable signal for "how many" emerges early, inside the vision encoder itself. But differences in overall counting accuracy between models trace back to the language model bolted on top, not to how well each model sees.
Counting is one of the more basic things human infants can do before they can talk, and most existing computer-vision counting tests don't isolate it, they tangle it up with correlated cues like object size or clutter. This study argues the raw ability to sense numerosity already exists early in these models' vision encoders, so the bottleneck is what the language model does with that signal afterward. That's a useful clue for anyone trying to fix VLM counting errors: the fix probably belongs on the language side, not in feeding the model prettier training images.
Given how often chatbots still miscount fingers in generated images or items in a photo, quietly logging seven models' worth of counting failures across 10,800 synthetic images feels less like a curiosity and more like a diagnosis.