Vision models can watch a video and still not understand cause and effect.
Researchers introduced a framework called CATE - Connecting Actions and Their Effects - to test whether video models can do something humans do automatically: look at an action and understand what it produces, or look at an outcome and trace it back to what caused it. The team split this into two sub-tasks: Action Selection, a semantic-level matching problem, and Effect-Affinity Assessment, a finer-grained version of the same idea. They built several baseline models and ran them through both tasks. The models struggled. Humans outperformed them by a large margin.
That gap matters because action-effect reasoning is load-bearing for anything that learns from demonstration - robotic systems that watch a human cook and then try to replicate the steps, or planning agents that need to predict consequences before acting. The researchers found a silver lining: even without explicit supervision, models trained on CATE tasks picked up useful properties like object tracking and pose encoding, suggesting the framework could work as a self-supervised training signal for video representations.
The results are another reminder that impressive benchmark scores on standard video tasks don't translate cleanly to the kind of causal, sequential reasoning that real-world deployment demands - a gap the field has been circling for years without closing.