[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-rubench-tests-coding-agents-on-russian-language-tasks":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},4391,"rubench-tests-coding-agents-on-russian-language-tasks","RuBench Tests Coding Agents on Russian-Language Tasks","A new benchmark exposes how well AI coding agents handle real maintenance requests written in Russian, and catches a product silently swapping models mid-run.","A research team has released RuBench 1.0, a 25-task benchmark that tests AI coding agents against repository-level bugs specified in Russian — and found at least one product quietly routing tasks to a different model than advertised.\n\nThe benchmark pulls fix commits from five live open-source projects — aiohttp, aiogram, Laravel, NestJS, and Fastify — covering Python, PHP, TypeScript, and JavaScript. Each task is written from scratch in Russian in the style of a real customer request, not machine-translated from English. All 25 commits postdate the training cutoffs of every model tested, which gives a reasonable contamination argument. The researchers evaluated Claude Code paired with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5, running each configuration three times. The best configuration resolved 78.7% of tasks.\n\nMost existing repository-level benchmarks assume English task specifications and curated issue formatting — conditions that rarely reflect how engineers outside North America and Western Europe actually file bugs. RuBench tests the gap between benchmark conditions and production reality. The finding that matters most, though, is not the pass rates: while auditing a fifth configuration using Fable 5, the researchers found that on 5 of 25 tasks the product silently fell back to Opus 4.8, meaning the thing being benchmarked was the deployed product stack, not the model in the headline.\n\nThat last detail should make anyone publishing model-vs-model leaderboard comparisons pause — you may be measuring infrastructure decisions, not model capability.","[\"ai\",\"benchmarks\",\"coding-agents\",\"open-source\"]","2026-07-08T04:00:00.000Z","2026-07-08T08:02:45.639Z","2026-07-08T08:02:48.597Z","published",null,[],"ai",[24,26,27,28],"benchmarks","coding-agents","open-source",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.06411",0,{"sections":35},[36,40,45,50,55,60,65,70,75,80,85,89,94,99],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":41,"slug":42,"count":43,"latest_published_at":44},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":46,"slug":47,"count":48,"latest_published_at":49},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":86,"slug":87,"count":83,"latest_published_at":88},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":100,"slug":101,"count":102,"latest_published_at":103},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]