A flaw in how AI agents evaluate their own skills can quietly disable the safety net that keeps them from getting worse over time.
Researchers studying self-evolving agents — systems that build and retire skills based on performance — found that a biased judge does not just add noise to the process. When the judge misclassifies failures as passes (called false-pass bias), it crosses a sharp threshold beyond which skill retirement stops working entirely. The team demonstrated this through a corrupted-reward analysis and behavioral experiments on a reference-free report-writing testbed, using code generation as a cross-check. Crucially, symmetric noise left retirement intact; only the one-directional false-pass bias broke it.
The dangerous part is the silence. The disabled retirement mechanism does not show up in aggregate performance metrics, meaning an operator could deploy a system that looks fine on paper while its skill library quietly drifts. The researchers note the downstream outcome is regime-dependent — evaluation quality degrades only where the same corruption also starves skill synthesis — but the structural failure is universal across domains and failure rates.
The paper proposes a practical fix: a defect-injection audit that tells operators, before deployment, which side of the threshold their judge sits on. That is a genuinely useful framing, since the problem is most acute for reference-free tasks where no ground-truth answer exists and LLM judges are the only option. Self-evolving agent frameworks have attracted serious research and commercial interest, but this work is a reminder that the feedback loops powering them carry assumptions that are easy to violate and hard to notice.