A new research testbed called LACUNA reveals that most AI unlearning methods don't actually delete sensitive information - they just hide it.
Researchers built LACUNA to test a specific claim at the heart of the unlearning field: that so-called "localize-then-unlearn" methods target and remove the exact model weights storing sensitive data. To check this, they injected personally identifiable information about synthetic individuals into known, predefined parameters of 1B and 7B OLMo-based models, then ran state-of-the-art unlearning methods against them. The results were damaging to the field's self-image. Despite strong performance on output-level tests - the kind existing benchmarks use - nearly all methods failed to hit the right weights and remained vulnerable to resurfacing attacks that recover the supposedly deleted data.
The gap matters because unlearning is increasingly being pitched as the fix for a real compliance problem: language models memorize training data, including private information, and regulators are starting to ask what companies plan to do about it. If the leading methods only suppress outputs rather than erase knowledge, that gap becomes a liability - legally and technically. LACUNA is the first benchmark that can actually tell the difference, by giving researchers a ground truth at the parameter level, not just the behavior level.
The encouraging detail buried in the paper: when localization is precise, even a simple gradient-based method achieves strong, attack-resistant erasure. The bottleneck isn't the deletion step - it's finding the right thing to delete. LACUNA is being released publicly, which should pressure the field to stop grading itself on output tests alone.