A new benchmark called LongVQUBench reveals that today's best vision-language models can't reliably judge video quality when the footage runs long.
Researchers built LongVQUBench to fill a gap they say existing tests ignore: most video quality benchmarks test short clips with isolated, obvious problems. This one covers over 1,200 videos - movies, documentaries, surveillance footage, egocentric recordings, and animation - paired with 1,500 questions ranging from multiple-choice to open-ended. The benchmark runs evaluations at three levels of difficulty, from spotting a localized glitch to reasoning about cumulative degradation across an entire long video. A fourth layer, called needle distortion question-answering, hides sparse spatial or temporal artifacts inside footage to test whether models can detect subtle problems at all.
The results matter because video understanding is one of the loudest selling points for multimodal AI right now, and this is evidence the pitch outpaces the reality. All 14 state-of-the-art models tested got measurably worse as video length and reasoning complexity increased - meaning the long-form video comprehension that labs have been demoing in highlight reels is shakier than advertised when quality evaluation enters the picture.
This mirrors a pattern from language models: benchmarks built on short, clean inputs consistently flatter model capabilities that fall apart on messier, longer real-world content. LongVQUBench won't fix that, but it at least gives researchers a consistent ruler to measure how far behind the marketing these models actually are.