A new benchmark says AI vision models trained to understand video are much worse at reading a fixed security camera than a handheld one.
Researchers built VANTAGE-Bench to test how vision-language models handle "Infrastructure AI" - the fixed-camera footage used for warehouse monitoring, traffic cameras, and building security, as opposed to the phone and dashcam-style clips most benchmarks use. They evaluated 17 models with no fine-tuning across 3,346 video and image assets spanning logistics, transportation, and smart-spaces footage, testing eight task types including dense captioning and object tracking. On general video question-answering and locating objects in a single frame, the models performed about as well as they do on existing benchmarks such as VideoMME and BLINK. But on tasks that require reasoning about when something happened - event verification, referring expressions, and temporal localization - they scored 9 to 24 points lower than those baselines. Temporal localization was the weakest skill in absolute terms too: no model in the test, at any scale, topped 55.7 mIoU, a low ceiling separate from the point gap above and a sign the task itself is unsolved, not just that these models missed it.
That distinction matters. A model that misreads a single photo is a curiosity. A model that cannot tell when an event started or ended is useless for the job infrastructure cameras actually do: flagging incidents, logging dwell times, tracking objects across a scene. The benchmark also found that on short-horizon object tracking, top models came within roughly 5 points of dedicated tracking software, but fell further behind as the tracking window stretched out.
One wrinkle worth noting: open-weight models led outright on 2D object localization, ahead of proprietary systems, which undercuts the usual assumption that more scale or a closed lab's secret sauce would close this gap on its own.