Researchers built a simulator to see if an AI copilot can actually diagnose a broken underwater robot, and the results show the choice of language model matters more than anything else.
The team describes SPAR (Simulation Platform for AUV Recovery), a testbed that pairs a real-time autonomous underwater vehicle (AUV) control system with a large language model that steps in only when onboard sensors flag something outside normal limits. They ran 480 trials, all simulating one type of failure: a mass shift that throws off the vehicle's center of gravity. Across those trials they varied the specific fault conditions, prompt structures, and mission scenarios, then tested one frontier LLM against three smaller models that can run locally on the vehicle's own hardware, with a separate LLM judging each diagnosis.
The frontier model ranked the true cause of the fault among its top three guesses in 85-90% of trials; the best local model managed 60-78%. That is a meaningful gap for a robot with no satellite link and no human to call. Weaker models often jumped to blaming a faulty control surface instead of finishing the full diagnostic procedure, a shortcut that looked plausible but was wrong.
The real news here is not the accuracy numbers, which cover a single fault type and a four-model roster and should not be read as a general verdict on AI's readiness for undersea repair work. It is the framework itself: a reproducible way to test whether LLMs reason correctly under uncertainty rather than just sound confident. Expect more purpose-built evaluation harnesses like this before anyone trusts a language model alone at the bottom of the ocean.