Giving an AI agent access to a scientific simulator does not simply make it smarter — it also introduces new ways to fail.
Researchers have released PHREEQC-MCQ-200, a 200-question benchmark that tests AI agents on aqueous-geochemistry simulations using PHREEQC, a widely used open-source chemistry modeling program. The benchmark requires agents to build simulator inputs, run the tool, read the outputs, and commit to a multiple-choice answer. Tested across several frontier and mid-tier model families, tool access raised average accuracy overall — but the improvement was not clean. Agents that gained points in some areas lost them in others, posting regressions that a single aggregate score would never reveal.
The finding matters because most evaluations of scientific AI stop at "did accuracy go up?" This benchmark surfaces something more useful: a tool-augmented agent can fail in ways a tool-free agent never would. A structured table-of-contents interface for reading simulator output, for example, cut token costs and helped stronger models — but hurt mid-tier models that could not reliably navigate it.
The broader point is that plugging an LLM into domain software is not a capability upgrade by itself; it is a new configuration with its own failure modes. As labs race to connect AI to lab instruments, simulation engines, and scientific databases, a benchmark that reports item-level retention and trajectory failures — not just a headline accuracy number — is exactly the kind of tool the field has been missing.