An AI agent built to read nuclear regulatory documents hits 81.5% accuracy on a benchmark most retrieval systems fail to crack.
Researchers tested an LLM-guided planning agent against a 200-question benchmark derived from NuScale's Final Safety Analysis Report, a document set spanning tens of thousands of pages. The agent works by observing evidence collected so far, choosing the next document fragment to inspect, and stopping when it judges the evidence sufficient. It maintains a dynamic knowledge graph as state and uses browse, read, and search tools against a structured document tree — no vector index required. The system scored 81.5% accuracy with a RAGAS Faithfulness score of 0.93, a metric that measures whether answers are grounded in the retrieved source text.
The headline result is not the accuracy score — it is the 38-percentage-point gap over a baseline that uses the same document tree without state-conditioned planning (43.5% versus 81.5%). That gap isolates planning as the lever. The system also beats LightRAG, HippoRAG, and GraphRAG by margins ranging from 8 to 32 points, and matches RAPTOR without requiring any offline pre-processing — a meaningful cost advantage for document sets that change frequently.
Nuclear filings are a stress test tailor-made for multi-hop reasoning: a single compliance judgment can hinge on passages scattered across dozens of chapters. Whether regulators or operators would trust an AI auditing tool at 81.5% accuracy — leaving roughly 1 in 6 questions wrong — is a separate and harder question the paper does not answer.