Frontier language models are still bad at figuring out what a scientist actually wants before they start crunching numbers.
Researchers built SciConvBench, a benchmark that tests how well LLMs handle multi-turn clarification in computational science tasks - fluid mechanics, solid mechanics, materials science, and partial differential equations. It grades models on two skills: pulling missing details out of a vague request, and catching requests that contradict themselves. The results are unflattering. Models did reasonably well at catching contradictions, but even the best performer resolved only 52.7% of the disambiguation cases in fluid mechanics, and it often filled gaps with silent, ungrounded assumptions rather than asking the user.
That silent-guessing habit is the real story. It's the same failure mode that shows up when coding assistants confidently invent an API instead of asking which library you meant - except here the stakes are a fluid-dynamics simulation or a materials model, not a broken build. An assistant that quietly picks its own boundary conditions isn't saving you a step; it's handing you results built on assumptions you never agreed to.
SciConvBench doesn't test whether models can solve science problems - it tests whether they know when they don't understand the question yet. Given that most "AI scientist" pitches assume the model can be trusted to fill in the blanks, a coin flip's worth of disambiguation accuracy is a useful reality check.