Amazon introduces FEB-30 benchmark testing agent memory stores against known facts

Amazon researchers introduce FEB-30, a benchmark of 30 scripted fact histories turned into 147 documents, so the correct memory state is known after each document.

Amazon says that on a memory system built like Mem0 and Hindsight, stored facts and QA scores differed by 16.8 points on average. The benchmark also found that pass/fail verdicts disagreed on 21% of fact histories, and the system chose the right memory action only 20.1% of the time.

The paper argues that QA scores alone miss broken memory stores, because the benchmark checks the stored facts directly against ground truth.