RAG is so back!
Another paper showing that plain RAG keeps winning the agent memory comparison.
They show that many dedicated memory systems score below naive retrieval over the raw history on LongMemEval, PersonaMem, and MINT-Eval, while costing much more.
They split a memory system's value into two parts. Memory benefit counts questions it answers that RAG misses. Memory tax counts questions RAG answers that the memory system gets wrong.
Why does the tax appear?
Building memory rewrites the history. Some facts get dropped or distorted during extraction, and some survive but become harder to retrieve. Memory systems also vary more than RAG when you change the backbone model.
A simple fix helped. Retrieving from the raw history alongside memory and reranking both together reduced the tax.
The practical takeaway here is that before adding a memory layer to your agent, compare it with plain retrieval over the transcript on your own data.
https://arxiv.org/abs/2610.34073
Chat with Paper: https://academy.dair.ai/papers/memory-tax-dissecting-the-net-value-of-agent-memory-2610.34073