Getting reliable benchmarks out of AI evaluators is harder than the labs make it sound.
Researchers expanded on earlier work to test 11 different configurations of LLM-based evaluation systems, measuring three properties at once: how tightly evaluators agree with each other (coupling), how varied their scoring strategies are (diversity), and how stable their scores are across small samples (reliability). The key finding is that optimizing for any two of these tends to wreck the third. Tightly coupled evaluators — those that agree with each other above a 0.9 correlation — scored reliably but produced nearly identical strategies, defeating the purpose of running multiple evaluators. Loosely coupled setups showed more diverse approaches but noisy, inconsistent scores.
This matters because the AI industry has increasingly leaned on LLM-as-judge setups to evaluate newer models — a practice that saves money compared to human raters but introduces its own measurement problems. If no evaluation configuration can simultaneously hit all three quality targets, then any benchmark built on these systems has a hidden structural flaw, and published results should be read with that in mind.
The study also flagged something odd: all four GPT-4o conditions showed zero coupling and maximum diversity across every test run — a pattern the authors attribute to version drift in the GPT-4o API during June 2026, meaning the model itself may have changed mid-experiment. That's a quiet reminder that "the same model" is not always the same model.