A new evaluation says an open-source AI research assistant is trustworthy for simple lookups but starts guessing once you ask it to connect the dots.
Researchers built AquiLLM, an open-weight, offline retrieval-augmented generation system meant to let astronomers query internal knowledge bases and legacy documentation in plain English. A team of domain experts tested the system's faithfulness, meaning whether its answers stay grounded in the retrieved documents instead of inventing claims, rather than just checking accuracy on generic quiz questions. Astronomy specialists found AquiLLM reliable on direct retrieval questions tied to specific documents, but its grounding degraded on tasks that required synthesizing multiple sources or resolving ambiguous queries. The paper argues that standard benchmark leaderboards miss this weakness entirely.
That gap matters because open and offline deployment has become the pitch for AI tools handling sensitive or proprietary research data, and astronomy has already lived through several generations of that infrastructure shift, from SQL-based archives to LLM interfaces. If faithfulness quietly degrades exactly when a researcher asks a harder question, that is the scenario most likely to produce a confidently wrong answer that ends up in a paper.
Swap a chatbot in for a database and you trade predictable emptiness for unpredictable confidence, which is a downgrade if nobody checks the citations.