Standard panels of AI judges have a bias problem that more judges can't fix.
The LLM Jury concept — where a panel of language models scores outputs and reports a consensus — has become a common alternative to relying on a single model as evaluator. Researchers have now shown, however, that any positive rate of contamination in that panel produces unbounded bias, regardless of jury size. The culprit: when even one judge fails in a characteristically LLM way — mode collapse, sycophancy, or reflexive safety refusals — it poisons the aggregate. Their proposed fix, RoPoLL (Robust Panel of LLM-as-Judge), swaps out the standard consensus aggregation for a geometric median, a robust mean estimator that is tuning-free and holds up until half the judges are compromised. Tested across 13 open-weight models ranging from 4B to 675B parameters, three benchmarks, and four corruption scenarios, RoPoLL outperformed standard panels on every biased corruption type — by roughly 19% on cross-dimensional attacks at equal compute.
The practical implication is significant for anyone using model-based evaluation at scale: the aggregation math matters as much as which models you pick. A three-judge RoPoLL committee using 38B-parameter models outperformed a single 675B-parameter model on one benchmark under 30% corruption — an 18x parameter advantage with better accuracy.
This lands at a moment when LLM-as-judge pipelines are proliferating across labs and product teams who want cheap, fast automated scoring. Most implementations still use simple averaging or majority vote, which this work shows is quietly fragile. The harder question is whether teams will audit their existing pipelines — or just assume the bias is symmetric and harmless.