[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-coding-agent-benchmarks-have-a-measurement-problem":10,"sections":40},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":35,"feedback":39,"feedback_at":22,"cost_usd":39,"total_tokens":39},3300,"coding-agent-benchmarks-have-a-measurement-problem","Coding Agent Benchmarks Have a Measurement Problem","A new audit finds that runtime instability and skewed scoring rules make three leading code-optimization leaderboards unreliable as progress signals.","Three popular benchmarks for coding agents may be measuring noise as much as progress.\n\nResearchers audited GSO, SWE-Perf, and SWE-fficiency by replaying official reference patches for 740 optimization tasks across four common types of Google Cloud machines. The results were shaky: reference patches passed the benchmarks' own validity rules in every cross-machine replay for only 39 of 102 GSO tasks, 11 of 140 SWE-Perf tasks, and 411 of 498 SWE-fficiency tasks. SWE-Perf was the worst offender — many of its reference patches produce close-to-zero runtime changes, which makes the baseline nearly meaningless. The team also found that SWE-fficiency's scoring rule assigns the ten hardest tasks weights of 58.5%-82.8%, inflating their pull on the final leaderboard score. Among the eight public submissions shared by GSO and SWE-fficiency, the two benchmarks disagreed on 9 of 28 pairwise comparisons — so which agent \"wins\" depends on which leaderboard you read.\n\nThis matters because leaderboard scores from these benchmarks are increasingly cited as evidence that coding agents are improving. If the benchmarks disagree with each other and fluctuate across machines, those citations are weaker than they look. The audit also found that at least one public submission already matches or beats the reference patch on 85.3% of replay-valid tasks, suggesting diminishing headroom for the benchmarks to differentiate agents.\n\nBenchmark fragility is a recurring problem in ML evaluation — Goodhart's Law tends to arrive quickly once a leaderboard becomes a proxy for capability. The authors stop short of calling the benchmarks useless; they propose using per-task reliability signals to filter out the noise. That is a reasonable patch, but it also means the raw leaderboard numbers need an asterisk.","[\"ai\",\"benchmarks\",\"coding-agents\",\"software-engineering\"]","2026-07-02T04:00:00.000Z","2026-07-02T06:53:56.759Z","2026-07-02T06:53:59.538Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The dek says the scoring rule assigns worst tasks weights 'up to 82.8%' but the body says 'up to 82.8%' while the source specifies a range of '58.5%-82.8%' — the body omits the lower bound, which is a partial suppression of a verifiable figure; additionally the body says the audit covered 'four classes of Google Cloud machines' but the source says 'four common types,' a minor paraphrase that is fine, however the body claims the scoring disagreement covers 'the same eight public submissions on GS","resolved","ai",[30,32,33,34],"benchmarks","coding-agents","software-engineering",[36],{"name":37,"url":38},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.01211",0,{"sections":41},[42,46,51,56,61,66,71,76,81,86,91,95,100,105],{"name":43,"slug":30,"count":44,"latest_published_at":45},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":47,"slug":48,"count":49,"latest_published_at":50},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":52,"slug":53,"count":54,"latest_published_at":55},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":57,"slug":58,"count":59,"latest_published_at":60},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":62,"slug":63,"count":64,"latest_published_at":65},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":67,"slug":68,"count":69,"latest_published_at":70},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":72,"slug":73,"count":74,"latest_published_at":75},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":77,"slug":78,"count":79,"latest_published_at":80},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":82,"slug":83,"count":84,"latest_published_at":85},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":87,"slug":88,"count":89,"latest_published_at":90},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":92,"slug":93,"count":89,"latest_published_at":94},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":106,"slug":107,"count":108,"latest_published_at":109},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]