A new benchmark shows vision-language AI models often pin the right attribute to the wrong object, and standard tests never noticed.
Researchers built InstaBind-Lite, a benchmark of 524 images containing 529 groups of three to six similar objects, 1,773 boxed instances, and 9,580 questions designed to isolate one specific failure. They call it Dense Same-Class Attribute Misbinding, or DSCAM: a model correctly detects that an attribute like a color is present in a scene, but glues it to the wrong instance of the same object type. Testing five open-source models and two commercial API models, the researchers found open-source systems misbound attributes 19.84% of the time on average, versus 7.55% for the API models. About 80% of those mix-ups involved objects sitting immediately next to each other.
This matters because it is invisible to the metrics teams already trust. Standard visual-question-answering accuracy just marks the answer wrong, and object-hallucination checks see both the object and the attribute in the image and call it fine. Neither approach tells you the model actually knows what is visible but not which thing owns which trait, a distinction that matters for anything from inventory photos to security footage where instances look alike.
The fix attempts tried so far, like forcing models to localize or identify instances before answering, helped some models and did nothing for others, which suggests this isn't one bug so much as a structural gap in how these systems bind vision to language.