[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-llm-judge-panels-share-errors-undermining-consensus":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},7280,"llm-judge-panels-share-errors-undermining-consensus","LLM Judge Panels Share Errors, Undermining Consensus","A new study shows LLM judges often fail on the same examples, meaning judge panels give far less independent confirmation than their vote counts suggest.","A new study finds that stacking LLM judges together does not multiply your confidence the way most benchmarks assume.\n\nResearchers tested ten LLM judges, spanning open-weight and frontier models, and measured how correlated their mistakes were. They found an average pairwise error correlation of 0.21, meaning judges tend to miss the same cases rather than fail independently. That correlation gets stronger, not weaker, among the highest-accuracy frontier judges, even ones built by different companies. The practical effect: those ten judges carry roughly the statistical weight of 3.5 independent judges, not ten.\n\nThat gap matters because panels are routinely used to decide which AI system performed better. The study found that in up to 28% of comparisons, ignoring shared errors makes one system look meaningfully better, a conclusion that disappears once the correlation is accounted for. The researchers also note that how errors are distributed, spread across most judges or clustered in a subset, changes which voting method actually works.\n\nTheir proposed fix is refreshingly low-tech: check judges against a small set of trusted examples first, map out where they share mistakes, and pick a voting method based on that before trusting the panel's verdict on anything new.","[\"llm-judges\",\"ai-evaluation\",\"ai-research\",\"benchmarking\"]","2026-09-23T04:00:00.000Z","2026-09-23T04:46:14.928Z","2026-09-23T04:46:19.256Z","published",null,[],"ai",[26,27,28,29],"llm-judges","ai-evaluation","ai-research","benchmarking",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.22512",0,{"sections":36},[37,40,44,49,54,59,63,68,73,78,83,88,93,98],{"name":38,"slug":24,"count":39,"latest_published_at":18},"AI",4247,{"name":41,"slug":42,"count":43,"latest_published_at":18},"Security","security",706,{"name":45,"slug":46,"count":47,"latest_published_at":48},"Policy","policy",369,"2026-09-23T02:13:52.000Z",{"name":50,"slug":51,"count":52,"latest_published_at":53},"Deals","deals",202,"2026-09-22T23:00:04.000Z",{"name":55,"slug":56,"count":57,"latest_published_at":58},"Hardware","hardware",168,"2026-09-22T23:56:03.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":18},"Science","science",133,{"name":64,"slug":65,"count":66,"latest_published_at":67},"Consumer Tech","consumer-tech",110,"2026-09-22T20:00:00.000Z",{"name":69,"slug":70,"count":71,"latest_published_at":72},"Software","software",80,"2026-09-22T23:32:52.000Z",{"name":74,"slug":75,"count":76,"latest_published_at":77},"Dev Tools","dev-tools",79,"2026-09-22T22:21:13.000Z",{"name":79,"slug":80,"count":81,"latest_published_at":82},"Startups","startups",65,"2026-09-22T22:06:48.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Gaming","gaming",45,"2026-09-22T15:35:06.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"General","general",43,"2026-09-21T23:48:56.000Z",{"name":94,"slug":95,"count":96,"latest_published_at":97},"Reviews","reviews",27,"2026-09-22T13:00:00.000Z",{"name":99,"slug":100,"count":101,"latest_published_at":102},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]