[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-llm-as-a-verifier-hits-782-on-swe-bench-without-extra-training":10,"sections":44},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":34,"tags":35,"sources":39,"feedback":43,"feedback_at":22,"cost_usd":43,"total_tokens":43},3729,"llm-as-a-verifier-hits-782-on-swe-bench-without-extra-training","LLM-as-a-Verifier Hits 78.2% on SWE-Bench Without Extra Training","A new verification framework treats LLM scoring as a probability distribution, hitting state-of-the-art on four benchmarks with no additional model training.","A research team is pitching verification — judging whether an AI solution is correct — as the next axis for scaling LLMs, and their framework backs the claim with numbers across four benchmarks.\n\nLLM-as-a-Verifier skips the usual approach of asking a model for a discrete score. Instead, it computes a continuous score from the distribution of scoring-token probabilities. That probabilistic foundation lets the framework scale along three levers: how fine-grained the score is, how many times the evaluation repeats, and how many sub-criteria the judgment is broken into. The result, the authors say, is better separation between good and bad solutions and more calibrated comparisons — without touching model weights. The team also built a cost-efficient ranking algorithm on top for picking the best candidate from a set.\n\nThe benchmark results are the headline: 86.5% on Terminal-Bench V2, 78.2% on SWE-Bench Verified, 87.4% on RoboRewardBench, and 73.3% on MedAgentBench — state-of-the-art on all four as of publication. Terminal-Bench V2 and RoboRewardBench are established evaluation suites for terminal-based agents and robotic reward modeling respectively; MedAgentBench covers medical reasoning. The framework also ships as a Claude Code extension, so developers can plug it into their own agentic pipelines to monitor task progress in real time.\n\nMost scaling research chases bigger models or more training data — this paper argues the verifier deserves its own scaling budget. If verification accuracy keeps improving as you throw more compute at it, it changes how teams should think about inference-time spending. That said, benchmark leadership is a crowded, short-lived title in AI research: treat these numbers as a snapshot, not a moat.","[\"ai\",\"benchmarks\",\"agents\",\"dev-tools\"]","2026-07-07T04:00:00.000Z","2026-07-07T06:52:39.129Z","2026-07-07T06:52:41.935Z","published",null,[24,30],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The headline and dek are not publication-ready: the title reads as a vague working placeholder ('A New Way to Check AI Work') rather than a precise, dry news headline — rewrite both to name the framework, the paper's core claim, and the concrete benchmark result, then verify that 'Terminal-Bench V2' and 'RoboRewardBench' are publicly documented benchmark suites before publishing, as neither name is independently verifiable from the source material alone.","resolved",{"id":31,"reviewer":26,"round":32,"reason":33,"status":29},"editor-r2",2,"The headline and dek now name the framework and cite the 78.2% SWE-Bench Verified figure, which resolves the vagueness concern, but [editor-r1] is not fully resolved because Terminal-Bench V2 and RoboRewardBench are named in the source abstract and the body omits them rather than verifying them — confirm both are publicly documented benchmark suites before publishing, or note them with appropriate attribution; additionally, the body drops the other benchmark results (86.5% Terminal-Bench V2, 87.","ai",[34,36,37,38],"benchmarks","agents","dev-tools",[40],{"name":41,"url":42},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.05391",0,{"sections":45},[46,50,55,60,65,70,75,80,85,88,93,97,102,107],{"name":47,"slug":34,"count":48,"latest_published_at":49},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":86,"slug":38,"count":87,"latest_published_at":18},"Dev Tools",59,{"name":89,"slug":90,"count":91,"latest_published_at":92},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":94,"slug":95,"count":91,"latest_published_at":96},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":98,"slug":99,"count":100,"latest_published_at":101},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":103,"slug":104,"count":105,"latest_published_at":106},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":108,"slug":109,"count":110,"latest_published_at":111},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]