A new benchmark finds that AI models built to understand video and audio together cannot reliably tell what happened when.
Researchers built AVTrace, a test suite of more than 34,000 training examples plus separate development and test sets, to check whether so-called omni models - AI that processes video, audio, and text together - can pinpoint when events start and end, keep them in order, and judge whether audio and video actually line up. They ran five existing open omni models through the tests using a scoring method that strips out reference text, so models cannot simply paraphrase their way to a good score. All five failed to beat a simple majority-guess baseline on the audio-video synchronization test, and all scored low on tasks that required stringing events into a sequence or reasoning about what one event implies for the next. One model, Qwen3-Omni-30B, got noticeably worse when researchers removed audio or video input or changed how images were processed, though the paper does not explain why; a smaller model, Gemma4-E4B-it, improved on several metrics after targeted post-training, with only modest and mixed effects on unrelated image benchmarks.
This punctures a quiet assumption behind a lot of AI benchmarking: that a model which describes a video accurately must also understand its timeline. AVTrace shows those are different skills - a model can generate text that sounds right without ever locating events correctly in time, which matters for anything from surveillance analysis to video-editing tools that claim to understand footage. The fact that fixes on one model did not cleanly transfer to unrelated tasks suggests these timing failures are not a simple bug to patch.
In an industry that markets omni models as all-purpose video understanding, a benchmark where every entrant flunks the basics is a useful reality check - and a testbed the next generation will actually need to pass.