A new diagnostic benchmark exposes a blind spot in how AI models understand first-person perspective when other agents are in the scene.
Researchers introduced EgoGapBench to isolate a specific ability they call Egocentric Action Selection — choosing the right action from your own viewpoint when other agents are visible. The distinction matters because existing egocentric benchmarks bundle first-person video data with perspective-taking ability, making it impossible to test the two separately. On EgoGapBench, humans answered reliably. Both open-source and proprietary multimodal large language models did not, and they showed a consistent failure mode: selecting actions performed by other visible agents instead of the one the model should be "doing" itself. Fine-tuning on existing egocentric data did not close the gap and sometimes made performance worse.
The gap between human and model performance here is not a quirk of dataset design — it suggests these models lack a stable sense of self within a scene. That has direct implications for robotics, AR assistants, and any deployment where an AI must act on behalf of a specific agent among many. A model that mistakes another actor's actions for its own is a liability, not a tool.
Fine-tuning on EgoGapBench's own training data did improve accuracy, but models still fell short of human-level performance — which means the benchmark is both useful and, for now, unsolved.