A new arXiv paper asks a deceptively simple question: when an AI agent seems to learn from a mistake, did it actually rewrite its own rules for success, or is it just reciting new words?
Researchers built a test called CMB-0.1 that requires five separate signals before crediting an agent with real criterion revision: the system has to catch its own failure, propose a fix, carry that fix into a new task, respond to a targeted intervention, and preserve the change over time. They ran it across twelve cross-domain cases and four different memory setups, from plain stateless calls to evaluator-controlled state, producing 192 model-case-arm trials. Not one trial satisfied all five conditions. One tested model, Qwen2.5-7B, answered transfer and preservation questions correctly even with zero actual revision state, meaning it looked like it had learned something it hadn't.
That gap matters because a lot of agent benchmarks quietly assume that if an agent's output improves after feedback, the agent must have updated its judgment. This paper shows that assumption can be wrong in ways current evaluation setups don't catch, and it matters more as companies ship agents meant to remember and adapt across sessions rather than just within one conversation. The authors are careful to say this zero result is about their measurement tool, not proof that no model can do this.
Their fix is a stricter follow-up protocol, CMB-0.4, with hidden tests and a frozen oracle grader. It is not a finished result, just a better ruler, which is a useful reminder that most claims about an agent learning from its mistakes right now are running on the old one.