Video-understanding AI has a guessing problem, and a new benchmark just made it hard to hide.
Researchers introduce STEMO-Bench, a benchmark of human-verified facts about objects in video, built specifically to catch multimodal large language models that get the right final answer for the wrong reasons. Instead of scoring only a model's last answer, which can be gamed by local visual cues or statistical priors, STEMO-Bench breaks each query into sub-questions that force a model to show its work on tracking object identity, state, and relationships across time. The same team also proposes STEMO-Track, an object-centric framework that builds structured object trajectories through chunk-wise state extraction and temporal aggregation, rather than treating each video frame as a fresh guess. In testing, that approach cut hallucinated answers and improved consistency compared with existing state-of-the-art models.
This matters because "final answer only" evaluation has been a soft spot in AI benchmarking generally, letting models look competent while actually pattern-matching. For video specifically, that gap is worse: a model can nail a multiple-choice question about a scene without ever tracking what happened to an object between frames. A benchmark that isolates that failure mode gives developers a real diagnostic instead of a vague accuracy score.
It is also a reminder that persistence, not perception, is the hard part of video AI. Recognizing an object in one frame is a solved problem. Knowing it is still the same object three seconds later is not.