[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-benchmark-finds-ai-models-struggle-to-translate-data-science-code":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},8295,"new-benchmark-finds-ai-models-struggle-to-translate-data-science-code","New Benchmark Finds AI Models Struggle to Translate Data Science Code","Claude Opus 4.6 topped a new ORCA benchmark but still got only 33.67% of full-project code translations right, and 56.92% on simpler ORCA-MAIN tasks.","A new benchmark shows that even top AI models struggle to translate data science code from one library to another without breaking it.\n\nResearchers built ORCA, a benchmark for what they call Data Science Code Translation: converting code between data science libraries while preserving its exact behavior. It has two parts. ORCA-MAIN covers 1,600 narrow tasks across data querying, data manipulation, and deep learning. ORCA-PROJECT covers 200 full, real-world data science projects across seven task types, with every translation checked against reference solutions and automated test cases.\n\nClaude Opus 4.6 topped the leaderboard, but its scores expose a wide gap between flashy code generation and the messier task of migrating existing code: 56.92% success on the narrow ORCA-MAIN tasks, and just 33.67% on full ORCA-PROJECT translations. Models did notably better when the source code spelled out each operation explicitly rather than relying on implicit shorthand, a pattern the researchers exploited by having models summarize a snippet's intent before translating it, which lifted scores by about five points on both benchmarks.\n\nEven with that intent-summarizing boost, getting a full project translation right remains roughly a 1-in-3 shot, closer to rolling a die than flipping a coin.","[\"ai benchmarks\",\"data science\",\"llm evaluation\",\"code translation\"]","2026-09-28T04:00:00.000Z","2026-09-28T22:19:49.117Z","2026-09-28T22:19:56.272Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"Fix the closing line's 'coin flip' framing — ORCA-PROJECT success is 33.67%, roughly 1-in-3, not a 50\u002F50 coin flip — and make the dek specify which benchmark the 'roughly half' figure refers to.","resolved","ai",[32,33,34,35],"ai benchmarks","data science","llm evaluation","code translation",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.30749",0,{"sections":42},[43,47,52,57,62,67,72,77,82,87,92,97,102,107],{"name":44,"slug":30,"count":45,"latest_published_at":46},"AI",4900,"2026-09-28T17:44:43.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":51},"Security","security",766,"2026-09-28T15:35:23.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Policy","policy",405,"2026-09-28T17:00:51.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Deals","deals",269,"2026-09-28T17:42:46.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Hardware","hardware",191,"2026-09-28T15:45:00.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Science","science",154,"2026-09-28T13:19:18.000Z",{"name":73,"slug":74,"count":75,"latest_published_at":76},"Consumer Tech","consumer-tech",139,"2026-09-28T17:09:47.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":81},"Software","software",91,"2026-09-25T20:55:00.000Z",{"name":83,"slug":84,"count":85,"latest_published_at":86},"Dev Tools","dev-tools",87,"2026-09-28T16:11:42.000Z",{"name":88,"slug":89,"count":90,"latest_published_at":91},"Startups","startups",80,"2026-09-28T17:50:28.000Z",{"name":93,"slug":94,"count":95,"latest_published_at":96},"General","general",49,"2026-09-28T16:44:57.000Z",{"name":98,"slug":99,"count":100,"latest_published_at":101},"Gaming","gaming",48,"2026-09-25T18:35:21.000Z",{"name":103,"slug":104,"count":105,"latest_published_at":106},"Reviews","reviews",31,"2026-09-28T14:31:34.000Z",{"name":108,"slug":109,"count":110,"latest_published_at":111},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]