Most year-over-year gains on LLM benchmarks may be measurement noise, not real progress.
Researchers analyzed 373,019 LLM-judge scores across benchmarks using generalizability theory, decomposing variance into system, item, judge, and interaction components. They found that pointwise rubric scoring, rating a single output on a scale, hits a hard reliability ceiling no matter how many test items you add, because judge-system interaction noise doesn't shrink with sample size. Switching to pairwise comparison, judging two outputs head-to-head in both presentation orders, cuts that noise by two orders of magnitude and pushes the ceiling to 0.986. But pairwise comparison introduces its own bias: a system shown first wins 8.6 percentage points more often than the same system shown second, a bias larger than the median improvement claimed across 53 published win-rate papers the researchers reviewed.
The measured noise floor, 0.41 to 1.24 points on a 0-5 scale, dwarfs the median reported gain of 0.28 points. On MT-Bench specifically, all 17 recovered improvements the researchers could match fell below the benchmark's own floor, and 70% of win-rate claims fell below the pairwise floor. A lot of leaderboard climbing, in other words, may not be real.
The kicker: an audit of 628 arXiv papers found fewer than one in four even state whether their evaluation was run more than once.