Researchers have built a diagnostic framework for the AI systems labs are using to automate scientific discovery — and the findings are less flattering to big-name models than the hype suggests.
AI-driven research systems pair large language models with automated evaluators to find algorithms, proofs, and designs without human-in-the-loop iteration. The problem: no one had a rigorous way to analyze why they succeed or fail. A new paper introduces GAMBLe, a framework that breaks each such system into four parts — generator, assessor, discovery mechanism, and budget — plus a combined object called the effective landscape that captures how the generator and assessor interact. The authors ran more than 760 replicated experiments across 46,000-plus iterations, testing configurations from single LLMs to adaptive ensembles on three NP-hard problems.
The uncomfortable result: there is no universal best setup. Frontier models underperformed open-source alternatives in multiple configurations, and the simplest search mechanism sometimes beat state-of-the-art meta-search. That matters because labs are currently scaling these systems without reliable tools to predict which component choices actually work — and the paper shows standard convergence guarantees do not hold under real ADRS conditions. Choosing the right components, even under tight budgets of 60 iterations, improved performance by 13-67% and search efficiency by 6-39x.
The broader implication: more expensive is not the same as more capable, and the field has been optimizing largely by intuition. GAMBLe is an academic framework, not a product, but if it gets traction it could give labs something they currently lack — a principled reason for the choices they are already making.