Three popular benchmarks for coding agents may be measuring noise as much as progress.
Researchers audited GSO, SWE-Perf, and SWE-fficiency by replaying official reference patches for 740 optimization tasks across four common types of Google Cloud machines. The results were shaky: reference patches passed the benchmarks' own validity rules in every cross-machine replay for only 39 of 102 GSO tasks, 11 of 140 SWE-Perf tasks, and 411 of 498 SWE-fficiency tasks. SWE-Perf was the worst offender — many of its reference patches produce close-to-zero runtime changes, which makes the baseline nearly meaningless. The team also found that SWE-fficiency's scoring rule assigns the ten hardest tasks weights of 58.5%-82.8%, inflating their pull on the final leaderboard score. Among the eight public submissions shared by GSO and SWE-fficiency, the two benchmarks disagreed on 9 of 28 pairwise comparisons — so which agent "wins" depends on which leaderboard you read.
This matters because leaderboard scores from these benchmarks are increasingly cited as evidence that coding agents are improving. If the benchmarks disagree with each other and fluctuate across machines, those citations are weaker than they look. The audit also found that at least one public submission already matches or beats the reference patch on 85.3% of replay-valid tasks, suggesting diminishing headroom for the benchmarks to differentiate agents.
Benchmark fragility is a recurring problem in ML evaluation — Goodhart's Law tends to arrive quickly once a leaderboard becomes a proxy for capability. The authors stop short of calling the benchmarks useless; they propose using per-task reliability signals to filter out the noise. That is a reasonable patch, but it also means the raw leaderboard numbers need an asterisk.