[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-ai-judges-score-differently-depending-on-what-language-you-ask-them-in":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},3616,"ai-judges-score-differently-depending-on-what-language-you-ask-them-in","AI Judges Score Differently Depending on What Language You Ask Them In","A new study finds that swapping the language of an AI evaluation prompt can flip which model ranks highest, raising questions about how we benchmark AI agents.","The model that wins an AI benchmark may depend less on its capabilities than on what language you used to write the test.\n\nResearchers ran 4,950 evaluation trials across five languages — English, Arabic, Turkish, Chinese, and Hindi — using six different AI judge models to score 55 coding tasks completed by three developer-agent frameworks. The central finding: backbone and language interact in ways that scramble the leaderboard. GPT-4o led in English with a 44.72% satisfaction rate, but Gemini outpaced it in Arabic (51.72%) and Hindi (53.22%). Agreement between judge models on individual requirements was modest, with a Fleiss' kappa at or below 0.231 — meaning judges frequently disagreed even when evaluated in the same language. Partial localization made things worse: translating the benchmark content but not the judge instructions cut Hindi satisfaction from 42.8% to 23.2%.\n\nThis matters because most agentic benchmarks default to English as a fixed evaluation language and treat that choice as neutral. It is not. If the judge's language changes which backbone appears to win, then published rankings may reflect evaluation setup as much as actual capability — a problem for anyone using leaderboards to pick production AI systems.\n\nThe researchers release their full requirement-level judgments and runtime statistics, which at least lets others check the work. Still, the broader implication is uncomfortable: the AI evaluation industry has spent years debating what to test, and comparatively little time asking what language the judge should think in.","[\"ai\",\"benchmarks\",\"evaluation\",\"multilingual\"]","2026-07-03T04:00:00.000Z","2026-07-03T09:41:35.365Z","2026-07-03T09:41:38.199Z","published",null,[],"ai",[24,26,27,28],"benchmarks","evaluation","multilingual",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.04532",0,{"sections":35},[36,40,45,50,55,60,65,70,75,80,85,89,94,99],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":41,"slug":42,"count":43,"latest_published_at":44},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":46,"slug":47,"count":48,"latest_published_at":49},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":86,"slug":87,"count":83,"latest_published_at":88},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":100,"slug":101,"count":102,"latest_published_at":103},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]