Persistent AI agents can recall what you told them last month — but erasing that data is harder than it sounds.
Researchers studying what they call "deployment-time memorization" found that AI agents increasingly store user information across sessions in external memory systems, not just baked into model weights. They tested three design levers — how aggressively the agent summarizes conversations, how many memory chunks it retrieves at once, and how it handles deletion requests — against two competing goals: personalization quality and how easily an adversary could extract private details. The key finding is that aggressive summarization cuts adversarial extraction rates by 64% to 76% depending on the model, while barely touching personalization quality. The catch: once that summary is written, deleting only the original raw text leaves the summarized copy intact and recoverable roughly 20% of the time.
This matters because "right to be forgotten" obligations under privacy laws like GDPR assume deletion actually works. If an agent summarizes what you said, then you delete the source, the summary — and the personal details inside it — can linger in a derived memory tier the deletion routine never touched. The researchers introduce a metric called the Forgetting Residue Score to quantify exactly that gap between what users think they erased and what the system can still surface.
The fix, per the paper, is either a full-pipeline purge that traces every derived copy or a tombstone redaction that flags deleted content across all tiers — neither of which most agent deployments currently implement. As AI assistants accumulate months of conversation history, the gap between "we deleted your data" and "we deleted all copies of your data" is going to become a compliance and trust problem worth watching.