Most teams validating an AI judge do not have enough overlapping human labels to trust the result, a new study finds.
Researchers measured how wrong-decision rates change as the share of items with multiple human labels, the overlap, shrinks. At just 5% pairwise overlap, wrong-deployment-decision rates hit 25%, and when picking the best judge among ten candidates, the odds of choosing the wrong one reached 65%. The team also derived a minimum-overlap threshold: 25% overlap is enough to reliably validate a judge that isn't a close call, though borderline judges stay hard to validate no matter how much data you throw at them. They tested the approach on 10 LLM judges across four evaluation setups, covering visual assessment, causal reasoning, and summarization.
Plenty of teams are quietly swapping an LLM in to grade another LLM's output, checking it against a handful of human ratings and calling it validated. This paper argues that sample is usually too small, and that how you draw it matters as much as how much you draw: a free stratified sampling scheme cut false-rejection rates in half compared to random sampling, when the strata were informative.
It's a math paper more than a product pitch, but if your evaluation pipeline runs on vibes and a few double-checked examples, the numbers here suggest you might be shipping on a coin flip you didn't know you were flipping.