AI agents deployed to handle real tasks can violate the very policies they are meant to enforce, and do it without triggering a single error message.
Researchers studying an airline booking benchmark found that in "policy-permissive" environments — where a tool executes any well-formed request regardless of whether that action is actually allowed — 78% of agent failures are silent wrong-state failures. A booking gets cancelled, a passenger count gets changed, a claim proceeds without required verification, and neither the tool nor the agent flags anything wrong. These results held across independent seeds, ruling out statistical noise. The fix the researchers tested is deliberately simple: four read-only pre-execution gates that inspect a proposed action against current state before permitting any write. On gpt-4o-mini, the suite raised task success from 29.6% to 42.0%, a gain that replicated on a separate seed set, with the improvement concentrated on the 26 of 50 tasks where the gates actually fired.
The result reframes a common assumption in agentic AI: that a more capable model is sufficient protection against policy violations. When tools are policy-permissive — which describes a lot of real integrations built for flexibility over constraint — the model's compliance depends entirely on the model playing nice, with no enforcement layer catching mistakes at the action boundary.
The researchers also report that a model identified in the paper as "gpt-5.2" — a name OpenAI has not publicly announced — showed the same failure pattern, but they flag this as "suggestive evidence, not a central claim," with five runs and no replication. Raw capability, in other words, did not make the problem go away.