A research team has a new method for helping AI agents figure out what they actually did, even when the video feed is full of distractions.
The problem is called agent ambiguity: when an AI learns from visual observations, the changes it sees between frames mix together its own movement, random background motion, camera shifts, and irrelevant objects. Latent Action Models — systems that infer action-like signals from observation transitions — struggle to untangle this mess without labeled data. The researchers introduce Observed Transition Factorization (OTF), which breaks each frame transition into a sparse set of reusable "primitives" representing discrete visual changes. Two model variants sit on top: OTF-LAM, which maps those primitives into action-like signals using a standard inverse-forward dynamics setup, and OTF-LAM-Dino, a decoder-free version that works in the frozen feature space of the vision model DINOv2.
The practical upshot is that downstream policy learning — the step where an agent actually learns to do something useful — matches or beats existing baselines even in scenes with heavy visual clutter. The primitives also transfer zero-shot across changes in the agent's body shape and what it's riding on, which suggests the representations are capturing something genuinely structural rather than overfitting to a specific visual setup.
Unsupervised action learning has been a persistent bottleneck for robot and game-playing agents that can't rely on labeled control data. This approach won't end that problem, but factoring out visual noise before abstracting actions is a cleaner architectural bet than trying to learn through the noise — and the DINOv2 hook means it can piggyback on representations the field already trusts.