[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-benchmark-finds-ai-agents-fail-most-real-engineering-software-tasks":10,"sections":40},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":35,"feedback":39,"feedback_at":22,"cost_usd":39,"total_tokens":39},8527,"benchmark-finds-ai-agents-fail-most-real-engineering-software-tasks","Benchmark Finds AI Agents Fail Most Real Engineering Software Tasks","A new preprint benchmark called EngiWorld tested seven frontier AI models on CAD, CAE, and other engineering software, and none topped an EngiScore of 44.3.","AI agents still can't run a CAD program end to end, according to a new benchmark.\n\nResearchers posted a paper on arXiv (arXiv:2609.37686), an unreviewed preprint published September 30, 2026, describing EngiWorld, a benchmark of 1,301 expert-curated tasks spanning six engineering domains: CAD, CAE, CAM, BIM, EDA, and 3D visualization. The tasks span 26 professional software platforms with both GUI and CLI interfaces, and cover six task types, from picking the right software to open-ended design work. The team built a verifier suite that checks whether an agent's output is geometrically valid, physically feasible, and rule-compliant, then scores partial credit against a design spec instead of just pass or fail. Seven frontier models were tested; the best scored an EngiScore of only 44.3, and multi-software attempts succeeded just 3.6% of the time.\n\nMost AI benchmarks measure chat or code generation, tasks with a keyboard and a text box. Engineering software demands geometry, physics, and file formats that don't forgive a rounding error, so a 44.3 score is a reminder that \"agentic\" still means something narrower in the physical-design world than in the browser.\n\nUntil an agent can open a CAD file, adjust a tolerance, and hand a clean file to CAM without a human checking its work, professional engineering shops have little to fear from automation.","[\"ai\",\"benchmarks\",\"engineering-software\",\"cad\"]","2026-09-30T04:00:00.000Z","2026-09-30T09:05:12.926Z","2026-09-30T09:05:19.374Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"Add explicit sourcing — name arXiv as the publication (paper arXiv:2609.37686), note it's an unreviewed preprint, and give the posting date\u002Ftimeframe, since the draft currently presents the benchmark's findings via vague 'researchers built' attribution with no named source or publication.","resolved","ai",[30,32,33,34],"benchmarks","engineering-software","cad",[36],{"name":37,"url":38},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.37686",0,{"sections":41},[42,45,49,53,58,63,68,73,78,83,88,93,98,103],{"name":43,"slug":30,"count":44,"latest_published_at":18},"AI",5028,{"name":46,"slug":47,"count":48,"latest_published_at":18},"Security","security",780,{"name":50,"slug":51,"count":52,"latest_published_at":18},"Policy","policy",417,{"name":54,"slug":55,"count":56,"latest_published_at":57},"Deals","deals",284,"2026-09-29T21:00:00.000Z",{"name":59,"slug":60,"count":61,"latest_published_at":62},"Hardware","hardware",194,"2026-09-29T13:16:04.000Z",{"name":64,"slug":65,"count":66,"latest_published_at":67},"Science","science",154,"2026-09-28T13:19:18.000Z",{"name":69,"slug":70,"count":71,"latest_published_at":72},"Consumer Tech","consumer-tech",142,"2026-09-29T18:38:03.000Z",{"name":74,"slug":75,"count":76,"latest_published_at":77},"Software","software",91,"2026-09-25T20:55:00.000Z",{"name":79,"slug":80,"count":81,"latest_published_at":82},"Dev Tools","dev-tools",89,"2026-09-29T17:15:00.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Startups","startups",83,"2026-09-29T21:51:36.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"General","general",49,"2026-09-28T16:44:57.000Z",{"name":94,"slug":95,"count":96,"latest_published_at":97},"Gaming","gaming",48,"2026-09-25T18:35:21.000Z",{"name":99,"slug":100,"count":101,"latest_published_at":102},"Reviews","reviews",31,"2026-09-28T14:31:34.000Z",{"name":104,"slug":105,"count":106,"latest_published_at":107},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]