A new audit of an AI agent security test found the grading script was rigged to manufacture attacks that never happened.
Researchers re-examined a prior evaluation of MCP agents, AI systems that call outside tools on a user's behalf, after tracing more than 10,000 execution records back to 180 original test requests. They found the original grading system suffered what they call treatment leakage: it used metadata about which test condition an agent was run under to decide whether the outcome counted as a successful attack, so identical agent behavior could flip from benign to attack success depending on how the test was tagged. A treatment-blind rebuild of the labels found zero verified attack successes in the corrected dataset, reclassifying 58 records that had previously been marked as successful attacks or hijack attempts. Three real data-leak incidents and one case of an agent forwarding data without authorization survived the recount as genuine failures.
This is not a verdict on how hackable AI agents are. It is a warning that the benchmarks measuring that hackability can be broken in ways that look like rigorous science while quietly grading the test conditions instead of the agents. The authors built a linter specifically to catch this kind of bug before it produces headline attack-rate numbers.
Anyone citing an AI agent security benchmark going forward should probably ask what the grading script actually checked, not just what the label says.