Amazon research finds on-policy distillation beats retrieval for agent user facts

Amazon researchers report that on-policy self-distillation stores user facts in model weights with 59.9% accuracy, compared with 36.9% for plain fine-tuning on the same model and user data.

Per a post by Elvis Saravia (@omarsar0), on eight HaluMem-Long users the distilled model, using no retrieved context, scores above MemOS-style retrieval (47.8%), extracted context (51.9%) and raw dialogue (51.3%). The post recommends keeping a text store for recent and missing facts and moving stable user knowledge into weights.