[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-ai-agents-still-stumble-in-messy-real-world-tests":10,"sections":48},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":38,"tags":39,"sources":43,"feedback":47,"feedback_at":22,"cost_usd":47,"total_tokens":47},3717,"ai-agents-still-stumble-in-messy-real-world-tests","AI Agents Still Stumble in Messy Real-World Tests","A new benchmark called AgentGym2 exposes how poorly even top-ranked AI agents perform when conditions get noisy and tools aren't handed to them.","Current AI agent benchmarks are making models look better than they are.\n\nResearchers published AgentGym2, a new evaluation framework designed to test large language model agents under conditions that more closely resemble actual deployments. Unlike most existing benchmarks, it does not hand agents pre-packaged tool interfaces or clean, fully specified inputs. Instead, agents must discover tools through exploration, compose them for tasks they haven't seen before, and stay on track when information is incomplete or noisy. The paper tested 15 proprietary and open-source models and found that even top-ranked systems struggled significantly — including Google's Gemini and what the benchmark authors labeled as OpenAI's top-tier model, though that specific labeling could not be independently verified against publicly documented release lineups.\n\nThe gap matters because the AI industry has been benchmarking its way to optimism. When evaluations assume clean inputs and pre-wired tools, they measure a laboratory version of the problem, not the one companies actually encounter when they ship agents into production. AgentGym2's results suggest the real-world performance ceiling is considerably lower than leaderboards imply.\n\nThis follows a familiar pattern: a new, harder benchmark arrives, top models stumble, labs announce they're working on it, and the cycle restarts. The more interesting question is whether evaluation frameworks like this one will influence how models are trained, or just how they're marketed.","[\"ai\",\"benchmarks\",\"llm-agents\",\"research\"]","2026-07-07T04:00:00.000Z","2026-07-07T06:36:15.534Z","2026-07-07T06:36:18.366Z","published",null,[24,30,34],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The dek references GPT-5 as a named model version, which cannot be verified against publicly documented release lineups as of publication — either replace with a verifiable model name or refer to it generically (e.g., 'OpenAI's top-tier model').","resolved",{"id":31,"reviewer":26,"round":32,"reason":33,"status":29},"editor-r2",2,"The dek avoids naming a specific model version, but the body still implicitly references GPT-5 via the source material's 'GPT-5' — more critically, the source abstract names 'GPT-5' as a tested model, and while the article body avoids the name, concern [editor-r1] was raised about the dek's prior reference; the current dek is clean, but the source itself names an unverifiable model version (GPT-5 is not on OpenAI's publicly documented release lineup as of publication), so the article must not pr",{"id":35,"reviewer":26,"round":36,"reason":37,"status":29},"editor-r3",3,"The source abstract explicitly names GPT-5 as a tested model, and the article reproduces this finding ('15 proprietary and open-source models, including top-ranked systems like Gemini') while implying completeness — but GPT-5 cannot be verified against OpenAI's publicly documented release lineup as of publication; either drop the source or refer to any unverifiable model only generically (e.g., 'OpenAI's top-tier model') and note that the benchmark authors' model labeling could not be independen","ai",[38,40,41,42],"benchmarks","llm-agents","research",[44],{"name":45,"url":46},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.05174",0,{"sections":49},[50,54,59,64,69,74,79,84,89,93,98,102,107,112],{"name":51,"slug":38,"count":52,"latest_published_at":53},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":55,"slug":56,"count":57,"latest_published_at":58},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":63},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":65,"slug":66,"count":67,"latest_published_at":68},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":70,"slug":71,"count":72,"latest_published_at":73},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":75,"slug":76,"count":77,"latest_published_at":78},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":80,"slug":81,"count":82,"latest_published_at":83},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":85,"slug":86,"count":87,"latest_published_at":88},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":18},"Dev Tools","dev-tools",59,{"name":94,"slug":95,"count":96,"latest_published_at":97},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":99,"slug":100,"count":96,"latest_published_at":101},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":103,"slug":104,"count":105,"latest_published_at":106},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":108,"slug":109,"count":110,"latest_published_at":111},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":113,"slug":114,"count":115,"latest_published_at":116},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]