A new study finds AI models judging other AI models play favorites - literally preferring outputs from their own model family.
Researchers tested four open-weight model families - Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5 - in a fully crossed pairwise evaluation totaling 9,312 judgments, the kind of setup increasingly used to grade chatbot outputs without paying humans to do it. A simple per-family scoring method turned out to be nearly indistinguishable from how good a candidate model actually is, correlating with model ability at r = 0.95, so the team built a corrected estimator that holds candidate quality fixed and isolates judge behavior alone. Once quality was controlled for, all four families still showed a same-family bias of 3.4 to 8.4 percentage points, for a global preference score of 0.067 (95% CI 0.053-0.084, p = 0.0002). The bias survived panel-based quality checks, a human-consensus baseline, and a separate replication run in float16 precision.
That matters because LLM-as-judge panels have quietly become the backbone of many leaderboards and internal eval pipelines that decide which model looks best. This study found that swapping panel composition alone flips 18.5% of pairwise outcomes relative to a family-balanced baseline - meaning who you pick as judges shapes the scoreboard almost as much as the models being judged. A second, compounding issue: simply reordering which answer comes first flips the verdict in 55.4% of AB/BA pairs, far more than random noise should produce.
Put together, this suggests a fair number of published model rankings say as much about judge selection and question order as they do about which model actually wrote the better answer.