The software wrapping your AI benchmark may be rewriting what the agent thinks is happening.
Researchers introduced a diagnostic called a belief rollout, which traces an agent's internal reasoning across multiple steps - covering its sense of progress, risk, recoverability, and likely next actions. When they held the task, environment, and underlying model constant but swapped the surrounding harness, the agent's beliefs shifted. Blocked actions, compressed repair logs, selective state verification, and evidence pruning all changed how the agent reasoned mid-task, even when the final success rate stayed the same. The divergence was decomposed into two components: an arrival term capturing immediate shifts from interface changes, and a growth term tracking how those shifts compound over longer decision horizons.
This matters because agent evaluations are almost always reported as pass-fail rates on a fixed benchmark - a number that conceals how the agent actually got there. If two harnesses produce the same terminal success but different belief trajectories, comparing agents across those harnesses is comparing apples to something that just happens to look like an apple. The researchers also propose BIWM, a protocol that standardizes observations, logs censored decision branches, and runs risky actions in shadow to align belief trajectories across harness configurations - no model retraining required.
Agent benchmarking is already a contested space: critics have long noted that leaderboard scores often reflect prompt engineering and evaluation tricks as much as genuine capability. This work adds a subtler concern - that the harness itself is an uncontrolled experimental variable, not neutral infrastructure, and published scores should probably say so.