Researchers built a benchmark that catches AI legal agents lying to themselves, not just to you.
Most tests of AI legal tools only check the final answer: did the bot cite the right case, yes or no. LexAgentHallu does something different. It tracks a legal agent's entire multi-step process, tool calls, reasoning chains, citations, and flags exactly where things break down. The benchmark has 3,414 test cases spanning 17 legal categories and 6 task types, each tagged against a taxonomy of 7 broad hallucination categories and 27 specific failure modes. The researchers ran it against 18 AI agents, both commercial and open-source.
The interesting finding isn't that these agents hallucinate. Everyone already assumes that. It's that they can land on the right answer through completely broken reasoning, what the researchers call a Right-Answer-Wrong-Reason effect. That should worry anyone using outcome-only grading to sign off on legal AI tools, because a correct citation today says nothing about whether the same process holds up on a harder case tomorrow. The study also found failure types cluster predictably by which agent framework and legal task is involved, suggesting these aren't random glitches but systematic weak points tied to how specific systems are built.
Legal AI has had a rough couple of years of headline-grabbing citation fabrications landing lawyers in front of angry judges. Most fixes since then have focused on better retrieval or stricter output filters, treating hallucination as a single symptom. A benchmark that localizes failure to a specific step in an agent's trajectory is a more useful diagnostic tool, assuming firms evaluating these products actually start asking for that level of detail instead of a pass rate.