A new benchmark says frontier language models are shaky at guessing the right settings for physics simulations, and slow to fix their guesses once they're wrong.
The benchmark, called SimulCost, tests LLMs on 11 physics simulators covering fluid dynamics, solid mechanics, and plasma physics, plus a twelfth production plasma code scored separately. Across 2,643 single-round tasks, where a model gets one shot at picking simulation parameters, frontier models succeeded 45-62% of the time, falling to 34-50% when the task demanded high accuracy. Letting models revise their guesses through trial and error, across 2,304 multi-round tasks, pushed success up to 66-81%. That improvement came at a cost: even in multi-round mode, the models ran 1.5-2.7x slower than a traditional scanning approach that simply sweeps the parameter space.
That tradeoff undercuts a common pitch for AI research agents: that they can replace tedious manual parameter tuning outright. SimulCost's numbers suggest LLMs are only worth deploying here once they are both more accurate on the first guess and cheaper than brute-force search, not just accurate eventually.
A model that gets the physics right on the third try, slower than a for-loop, is not much of a shortcut.