Picking the right frames from a video turns out to matter more than anyone benchmarked.
Researchers have released VG-GUIBench, a benchmark that tests whether AI agents can watch a video tutorial and then complete the same task on a real graphical interface. Alongside it, they introduce TASKER, a keyframe extraction algorithm that selects frames by weighing both task relevance and scene change. On two established video question-answering datasets, TASKER beat the previous best method by 2.0% on EgoSchema and 1.8% on NExT-QA. Code and data are public on GitHub.
The gap this closes is real. Most video AI benchmarks check whether a model spotted something in a clip. VG-GUIBench asks whether a model learned a procedure and can execute it — a much harder bar that reflects how people actually use tutorial content. Keyframe selection, it turns out, is the chokepoint for both tasks.
Multimodal models have raced up leaderboards on video question-answering, but those leaderboards mostly reward shallow visual recognition. A 2% gain from better frame selection suggests the field may have been optimizing around the wrong bottleneck all along.