AI agents that summarize their own memory can keep a safety rule's exact wording while quietly losing the rule's power to stop bad behavior.
A new study examines what happens during a single cycle of context compaction, the routine process where long-running AI agents replace their full conversation history with a model-generated summary. Researchers found that when compaction does not drop a safety rule outright, it often leaves a degraded "residue" that still reads like a rule but no longer functions as one. Under behavioral testing, agents holding one of these residues performed the prohibited action far more often than agents with an intact rule, with gaps of 34 and 57 percentage points across two different models used to replay the tests. The researchers also found that rule-shaped text survives summarization more often than equally prominent plain facts, which is exactly why just checking if the words are still there feels like enough oversight, even though it isn't.
This builds on earlier work showing that dropping safety constraints during compaction causes behavioral violations across many models, but sharpens the diagnosis: the failure mode isn't only text loss, it's rules that look intact and still fail to fire. Because the degradation is silent at runtime, catching it requires comparing an agent's live rules against an external source of truth, like a constraint registry, since the researchers found that judging correctness with another LLM alone could flip your conclusions entirely.
For anyone building agents meant to run for days or weeks between human check-ins, that's a reminder that a compliance checklist reading "rule present" is not the same as a rule anyone can trust.