A new benchmark says today's AI models still can't reason about what a factory robot is actually doing.
Researchers built FactoryBench, a test set of more than 70,000 question-and-answer pairs drawn from roughly 15,000 sensor-data episodes recorded on a UR3 cobot and a KUKA KR10 industrial arm, plus two existing datasets, AURSAD and voraus-AD. The questions are split into four tiers modeled on Pearl's ladder of causation: basic state, intervention, counterfactual reasoning, and decision-making. Four of the five answer formats are graded by fixed rules, while free-form answers get scored by a panel of LLMs voting on quality. The team then tested six frontier LLMs against the benchmark with no task-specific training.
None of the six models topped 50% accuracy on the structured, lower-level questions, and performance collapsed to under 18% once the task shifted to deciding what a robot should do next. That gap matters because industrial automation keeps getting pitched as a place for AI copilots to monitor and troubleshoot machinery in real time.
Reading sensor dashboards, it turns out, is a different skill than reading the internet - and right now these models haven't learned it.