A new tool turns large language models into bug-hunters for the software that runs machinery, cars, and robots.
Researchers built LLM-Falsifier, a system that searches for cases where a cyber-physical system violates its formal specification, written in a math notation called Signal Temporal Logic. Instead of using traditional numerical search algorithms, it prompts an LLM repeatedly, feeding it not just numbers but natural-language variable names, full output trajectories, and the exact moments where the system came closest to failing. Tested on the ARCH-COMP benchmark suite, a standard proving ground for this kind of tool, it needed fewer simulation runs than existing methods on 14 of 21 specifications.
Falsification tools have long relied on purpose-built optimizers: Bayesian search, surrogate models, search-based testing. The fact that a general-purpose LLM, handed extra context a numerical solver would ignore, can out-search several of those specialized tools suggests the advantage isn't raw computation but interpretation - the model appears to reason about what the trajectory data means, not just crunch it.
Still, it lost on 7 of the 21 specifications, and the paper measures success in simulations needed, not wall-clock time or dollars - an LLM call is far slower and pricier per query than a Bayesian optimizer, so "more sample-efficient" isn't the same as "cheaper" yet.