You do not need to rerun an entire benchmark every time an AI agent gets an update. A fraction of the questions will do.
Researchers studied a production analytics agent used by tens of thousands of people every month, and pulled 574 historical benchmark runs, splitting them chronologically into a calibration set and a held-out test set. They compared four evaluation strategies: random sampling, caching old results, fixed representative subsets, and adaptive testing based on item response theory. The adaptive method won on paper: a 200-question run, just 38.5 percent of the full benchmark, matched full results within about 1 percentage point of error. But the team shipped a simpler approach instead, difficulty-stratified fixed subsets, which needed no recalibration to work on five other agent families and stayed accurate even when the calibration window shrank to a single day.
This is the unglamorous problem every team shipping agents eventually hits: benchmarks that take an afternoon to build take an afternoon to rerun every time the model changes, and most teams do not have that afternoon to spare. The tradeoff here, picking the second best statistical method because it is dramatically easier to operate, is the more useful finding, since it matches the decision most engineering teams will actually make.
The best method on a spreadsheet is not always the one that ships, and this paper is refreshingly honest about choosing convenience over the extra decimal point of accuracy.