NVIDIA Research and Inferact present REINFORCE Zero for stable asynchronous RL
AINVIDIA Research and Inferact present REINFORCE Zero, a recipe for stable asynchronous reinforcement learning with importance sampling. The paper reports that aligning rollout and training kernels removes the fluctuating mismatch, letting unclipped importance sampling correct the remaining weight lag and keep training stable for 20,000 updates.
AIWhy it matters.
The post separates weight lag from kernel mismatch in asynchronous RL and shows how aligning kernels lets plain unclipped importance sampling stay stable.
Source details
vLLM Blog · Ao Shen (Inferact), Jian Hu (NVIDIA Research), Yan Bai (NVIDIA DevTech), Shekhar Pandey (AMD), and the REINFORCE Zero authors · vllm.aiPublished · added here
Original title: Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1