A new paper argues that a popular way of benchmarking AI code-search tools is quietly broken.
The approach in question is LLM-driven evolutionary search: an AI model generates a batch of candidate programs, called seeds, then iterates on them to improve a score. Researchers testing these systems typically pick one setup, say one seed run for a fixed number of iterations, and declare a winner based on that single result. A new study tested three such strategies across five standard optimization tasks, but instead of picking one setup, ran the full grid of seed counts and iteration counts. The result: the best split between running more seeds versus more iterations changes depending on the strategy, the task, and the total compute budget. On one task, the worst-performing strategy at a single seed became the best strategy at forty seeds. On another, adding more iterations past a certain point actively wasted budget that would have scored higher spent on more seeds.
This matters because it means published rankings in this niche but growing area of AI research may simply reflect an arbitrary choice of test settings rather than which method is actually better. If a paper's headline result depends on where it happened to stop testing, other researchers building on that result are building on noise. The authors propose reporting a full seeds-by-iterations frontier instead of a single score, so comparisons actually hold up.
It's a small, unglamorous fix, but it echoes a familiar problem in machine learning research: benchmarks that look rigorous because they produce a number, even when that number depends heavily on invisible knobs nobody disclosed.