Google Research adds a "sleep" phase to continual learning for LLMs

AIGoogle Research says it has developed a "sleep" phase for continual learning that lets LLMs keep integrating new facts and improve reasoning. The method aims to reduce catastrophic forgetting and static knowledge cutoffs.

Overview of our sleep framework: Input streams from the active phase are stored in short-term memory before entering the sleep cycle, which bifurcates into Consolidation (transferring fragile short-term tokens into expanded long-term parametric memory using low-rank structures and MoE) and Dreaming (autonomous self-improvement via synthetic routing with random experts and RL refinement).

Source details

Google Research · x.comPublished Oct 27, 1:40 PM ET

Original title: Today on the blog, learn about our new instance of a “sleep” phase in continual learning, which enables LLMs to continuously integrate ne...

Topics

Google / Gemini Reasoning Data & training Papers & research