vLLM blog: bitwise-identical rollout and training logprobs stabilize RL training
Original titleFinish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven
The vLLM blog reports that training and inference compute bitwise-identical logprobs for every sampled token, giving a train-inference KL of 0 at every step.
On DeepSeek-V4-Flash, a high-performance baseline's KL rose from 2e-3 to 29.9 while its reward fell back.
The consistent version kept 91.3% of baseline rollout throughput, and the authors packaged the process as an open-source agent skill that they also applied to Nemotron-3.5.
The post explains how bitwise-identical logprobs between rollout and training make RL training stable, with measured KL and throughput numbers that show the cost of this alignment.
Source: vLLM Blog · vllm.aiPublished · added here