A new study shows you can get a panel of AI judges to agree with each other more often - without getting them any closer to what a human would actually say.
Researchers tested a label-free method for fixing the rubrics that panels of AI models use to score things like summaries, health-related answers, and chatbot replies. The method splits a rubric into sub-criteria, flags the ones where judges disagree most on a training set, and has a separate model rewrite only those contested clauses - human labels are never used to pick or edit anything, only to check the result afterward. The team ran 129,770 judge requests using five judges from three providers across three benchmarks: SummEval, HealthBench, and USR Topical-Chat. On SummEval, the edited rubric raised inter-judge agreement (Krippendorff's alpha) by 0.069 over the original rubric and by 0.074 over a control that edited random, uncontested clauses instead.
That jump in agreement didn't make the judges line up with human experts any better - the change in agreement with human ratings was statistically indistinguishable from zero. On HealthBench and USR Topical-Chat, the fix didn't even move inter-judge agreement. In a secondary comparison, editing the contested clauses left judges slightly less aligned with human raters than editing random clauses did, on all three datasets, though none of those gaps was statistically conclusive.
So the common practice of judging rubric quality by how often AI judges agree with each other quietly skips the one check that actually matters: whether they agree with us.