A research framework designed to catch AI hallucinations turns out to be a reliable signal only for certain model families — and actively misleading for others.
Researchers introduced Evidence Graph Consistency (EGC), a method that maps the structural relationships between retrieved source passages and an AI-generated answer, then measures how well those structures align. Tested across 5,767 responses from six large language models on the RAGTruth benchmark, EGC worked as expected for Llama-2 models — high structural inconsistency correlated with hallucinations. But for GPT-4, GPT-3.5, and Mistral-7B, the signal flipped: higher graph consistency was associated with hallucinated answers, not accurate ones.
That reversal matters because it suggests model families hallucinate in fundamentally different ways. A detection tool calibrated on one family may not only fail on another — it may steer reviewers in the wrong direction. The broader implication is that retrieval-augmented generation, often pitched as a hallucination fix, still lacks a reliable, model-agnostic method for catching the errors it does not prevent.
The finding is a quiet rebuke to the field's habit of benchmarking on one model and claiming general applicability. EGC is a useful diagnostic, not a universal one.