AI agents that claim to plan chemistry experiments can't reliably figure out what the ingredients cost.
Researchers built ChemCost, a benchmark of 1,427 reactions tied to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes. To price a reaction, an agent has to identify the right chemicals, find valid purchasable packs from suppliers, convert units, and do the arithmetic. The team tested frontier models, open-weight models, and chemistry-specialized agents, then added a stress test: noisy versions of the same reactions with altered chemical names, odd quantity phrasing, missing fields, and messy formatting. The best agent hit only 50.6% accuracy within a 25% error margin on clean inputs, and performance dropped further once the noise was added.
This is a quieter failure than a chatbot inventing a chemical name - it's agents with real tools still blowing the bookkeeping: misreading a label, grabbing the wrong pack size, or never converging on an answer at all. Procurement cost estimation is about as mundane and necessary as lab tasks get, and if agents can't handle it reliably, the harder problem of running synthesis safely and economically is further off than demos suggest.
Tool access was supposed to fix LLMs that hallucinate facts. ChemCost is a reminder that handing a model a calculator and a supplier catalog doesn't mean it knows how to use either one.