Benchmarking a chatbot is expensive, and a new paper argues most labs are doing it inefficiently.
The method, called Speculative Evaluation, targets a real cost problem: because large language models are stochastic, benchmark scores are averages over many randomized rollouts, and running the same number of rollouts on every task wastes budget on tasks that barely vary while starving the noisy ones. The researchers' approach runs a short uniform pilot, pools per-task success counts using a hierarchical Bayesian model, then uses the resulting variance estimates to allocate the remaining rollout budget where it actually reduces uncertainty - a technique borrowed from classic Neyman sampling theory. An async variant, HBN-async, gets around the delay of waiting for the full pilot to finish before reallocating. Tested across six model checkpoints and 18 benchmark groups (107 profiles total), it cut variance by 12.8% to 33.6% compared to uniform sampling, at rollout budgets of 8 to 64 per task.
This matters because eval compute is not free, and sloppy variance is how leaderboards end up with rankings that flip on noise rather than real capability differences. A method that gets tighter confidence intervals from the same budget - or the same confidence from a smaller one - is a genuine efficiency win for anyone running large eval suites, not just academic labs with unlimited GPU time.
Worth noting: this is a statistics paper about measuring benchmarks better, not a new benchmark or a smarter model. It does nothing to fix benchmarks that are already poorly designed or gamed - it just measures whatever they measure with less noise.