AI models that analyze video often know when they're bluffing.
Researchers built TRAPSBench, a benchmark of 1,404 paired physics videos where a small change makes the outcome impossible to determine from the footage alone. Testing 16 vision-language models across five families, the best model scored just 0.292 on a new metric called Penalized Epistemic Calibration Score, which rewards correct answers when the outcome is knowable and abstention when it isn't. The models mostly guessed instead of admitting they couldn't tell. But probing their internal states told a different story: linear probes could predict whether a video's outcome was actually answerable with up to 0.91 AUROC, and researchers could flip a single internal "direction" to switch abstention on or off.
That gap matters because it reframes a familiar failure. AI hallucination is usually described as models not knowing what they don't know. This paper suggests that's often wrong, at least for vision. The information needed to say "I can't tell" is sitting right there in the model's hidden layers. Something between that internal signal and the words coming out is dropping the ball.
The researchers also found models are far better at flagging uncertainty in text than in video, catching textual impossibility about four times more readily than missing visual evidence. That tracks with how these models are trained: heavily on text, with vision treated more as an add-on. If a self-driving system or a video-based safety tool inherits this pattern, it's not a knowledge problem, it's a plumbing problem, and one that output-level fixes might actually solve without retraining the whole model.