AI judges scoring chatbot answers against rubric checklists give different verdicts depending on which other rubrics are graded alongside them, according to new research.
Researchers tested large language models used as judges, which increasingly grade responses against fine-grained rubric checklists instead of a single pass/fail call, since running each rubric as its own inference call is accurate but slow. Evaluating every rubric together in one pass is a tempting shortcut, but the team found it introduces what they call rubric interference: a verdict on one rubric shifts depending on which other rubrics are bundled with it. In preliminary tests, only about a third of samples got a fully consistent verdict on the same rubric across different combinations of co-present rubrics. To counter this without new human labels, the researchers built Self-Anchored Rubric Alignment (SARA), which treats a model's own single-rubric judgments as a stable anchor and trains the model to align its multi-rubric reasoning with those anchors through on-policy self-distillation.
This matters because the failure is invisible to standard accuracy scores - it is not about a judge being wrong, but about a judge disagreeing with itself depending on how a request is packaged. As rubric grading becomes the default way to certify chatbot quality, including in sensitive domains like health advice, that kind of instability undermines confidence in the pipelines used to sign off on models. SARA held up across three datasets (HealthBench, FLASK, ResearchQA) and two model families (Qwen3, Llama-3.1), and its consistency gains transferred to datasets it was not tuned on, while still agreeing with both the base models and GPT-4.1 as a reference judge.
The efficiency pitch - grade everything in one pass - turns out to have a hidden cost: batching questions for speed can quietly make graders less trustworthy unless someone trains that instability back out.