Researchers built a test to see if video AI models can recognize repeating patterns. They mostly cannot.
A team developed CycliST, a benchmark of synthetic video sequences designed to probe video language models on cyclical state transitions — objects moving in loops, colors shifting periodically, sizes scaling and shrinking on a cycle. The benchmark applies progressively harder conditions: more objects on screen, more visual clutter, worse lighting. The researchers tested a range of current state-of-the-art models, both open-source and proprietary, across this tiered task set. The results were consistent: models struggled across the board, failing to detect cycles, count moving objects, or extract basic quantitative information from scenes.
The more telling finding is what did not correlate with performance. Neither model size nor architectural choices predicted which model would fare better, and no single model led consistently across all task types. That closes the usual escape hatch — "just train a bigger model" — and points to something more structural: these systems lack a reliable notion of time. Cyclic patterns require a model to hold and reason about state across frames, not just match visual features to a prompt.
For a field where video understanding is increasingly sold as near-solved and essential for autonomous systems and robotics, being stumped by a looping animation is an uncomfortable data point worth sitting with.