New Tsinghua paper finds that when people ask AI agents with long-term memory to forget a personal detail, the assistants almost never remove it cleanly, either keeping it or deleting too much.
Agents with long-term memory store personal details, and users increasingly ask them to delete things like a home address. Existing memory benchmarks test only storing and recalling facts.
They built BEMULA, 50 long synthetic chats with 135 forget requests, each paired with a copy that has no request. Probes check whether the fact is gone, other memories survive, and attacks can still extract it.
No tested setup passed more than 1.5% of forget requests end to end, across 6 memory systems and 3 backbone models. After a forget request, systems blocked at most 35.7% of simple extraction prompts like role-play or yes-or-no guesses.
If your agent remembers users, build and test a real delete path, since a confirmation message does not mean the data is gone.