AI models that grade their own homework may be most dangerous when they're most consistent.
Researchers tested 10 frontier large language models across 491 concepts to measure what they call generator-evaluator self-consistency — whether a model applies the same concept the same way when producing an answer and then later judging it. The study, posted to arXiv, found wide variation across models on this metric. But the more troubling finding came from a clinical test set using physician-validated mistakes: models that scored higher on self-consistency were also more likely to fail to catch real errors.
This matters because the entire architecture of agentic AI pipelines rests on a shaky assumption — that a model checking its own output is a meaningful safeguard. If the model that generated a wrong answer is also the one flagging mistakes, and that model is consistent in its wrongness, you don't get a safety net. You get a confident rubber stamp. The researchers call this the "consistency dilemma": the property that makes self-evaluation operationally useful is the same one that makes it unreliable.
The AI industry has leaned hard into self-critique and self-refinement loops as substitutes for external verification. This research suggests that approach has a structural flaw that better benchmarks alone won't fix.