A new open-source tool called DeltaSelect trims coding agent benchmarks down to a small set of tasks that reliably predicts the full suite's score, aiming to make routine A/B testing cheaper.
The researchers first checked whether individual benchmark tasks are even useful signals. Resampling published trials from DeepSWE, they found only 22 of 113 tasks, about 19.5%, had a fifth-percentile Pearson correlation of at least 0.50 with full benchmark performance. DeltaSelect uses that same correlation approach to pick a fixed, budget-capped task set, then maps partial credit verifier results onto a common scale with linear regression. In a case study tuning custom skills and instructions for a low-reasoning version of gpt-5.6-luna, the team ran 13 evaluations for a total of $27.86. The version they ultimately adopted cost 58.1% less to test than their initial version, $1.75 versus $4.18, a statistically significant drop, and it scored higher on the calibrated benchmark, 42.36% versus 36.46%.
Most benchmark tooling is built for one-time model rankings, not the daily grind of comparing a baseline agent to a candidate change. DeltaSelect is explicitly not meant for the former. That distinction matters because teams iterating on prompts, skills, or fine-tunes need fast, cheap feedback loops, and running a full suite on every tweak is the kind of cost that quietly discourages experimentation.
Worth noting: the cost savings held up statistically, but the score improvement did not, with a p value of 0.326, a reminder that cheaper testing and better testing are not automatically the same thing.