A new benchmarking paper argues that the leaderboards used to rank AI coding agents have been measuring something close to noise.
The authors say popular implementations of the pass@k reliability estimator, first defined by Chen et al. in 2021, quietly plug in the wrong number: the count of unit tests in a single submission instead of the count of independent rollout attempts, a mixup they prove invalid by counterexample. Their fix, reliability@k, applies the same formula correctly, with n as independent rollouts and c as the rollouts that fully pass. On a synthetic multi-rollout benchmark, the misapplied metric reported scores of 0.96-0.98 where the corrected math produced just 0.00-0.12, an inflation of up to 0.97 in absolute terms. A cheaper single-rollout proxy, the kind teams use to cut compute costs, also failed to substitute for repeated runs, correlating at only 0.417 on Spearman's rank test.
That gap matters because agent benchmarks increasingly shape which coding tools teams buy and how vendors market them, and a small SWE-bench Verified pilot in the paper found the same pattern in a live repository: agents passed 80% of hidden tests on average but only fully resolved 20% of tasks outright. The authors also tried a security-adjusted version of the metric that discards rollouts containing high-severity insecure code, though in their three-agent trial it didn't move any rankings - a result they describe as inconclusive rather than reassuring.
Coding-agent marketing has been running ahead of the math meant to check it; a leaderboard number is only as honest as the estimator behind it.