A new paper argues that a lot of AI agent benchmarks may be scoring runs before they're actually finished.
Researchers built a framework for judging when an agent evaluation run counts as truly over, splitting the question into two separate conditions: whether the outcome itself is settled (outcome finality) and whether one run's state has been kept separate from the next (cross-unit separation). In a controlled replay, they held an agent's actions fixed and found that a delayed write - an operation that completes after the run stops but before the next one starts - can quietly change the score of the following run if the underlying service state persists between runs. Isolating runs or verifying a reset prevented the contamination. The team then reviewed ten public agent evaluation protocols and found that all of them define when a run stops and what gets scored, but few document whether background operations have actually finished or whether runs are truly isolated from each other.
That gap matters because agent benchmarks increasingly function as marketing copy - a headline completion rate implies a clean, repeatable measurement. If a scored outcome depends on whether a database write landed a few seconds late, or whether a previous run's leftover state bled into the next one, published numbers can overstate how reliable an agent actually is.
This is the flaky-test problem, just relocated from CI pipelines to AI leaderboards - software engineers solved shared-state contamination with test isolation and fixtures decades ago, and agent evaluation is now rediscovering why that discipline exists. The paper's proposed fix, an open-effects record that tracks what might still be pending when a run ends, is a reasonable start, though it will only help if benchmark maintainers actually adopt it.