Nearly half of the AI benchmarks used to measure model progress have hit a ceiling, making them poor tools for distinguishing today's best models from tomorrow's.
Researchers analyzed 60 language model benchmarks using 14 properties tied to saturation — the point where top models cluster so tightly at high scores that the benchmark stops being informative. They found saturation rates climb with a benchmark's age, which is roughly what you'd expect: a test written in 2019 wasn't designed to separate models that didn't exist yet. More usefully, they identified that expert-curation — not whether test data is public or private — is the key factor in how long a benchmark stays useful.
This matters because the AI industry leans heavily on benchmark scores to market models, guide enterprise buyers, and allocate research dollars. When the measuring stick is broken, everything downstream — deployment decisions, capability claims, funding pitches — rests on shakier ground. A model topping a saturated leaderboard isn't necessarily the best model; it may just be the latest one tested on an exam everyone has already aced.
The findings land as labs race to release ever-harder evals — ARC-AGI, FrontierMath, and others — to stay ahead of the saturation curve. The researchers suggest that thoughtful design choices at benchmark creation can extend longevity, which is a polite way of saying the field has been careless about building evals that last.