vLLM supports day-0 LoRA reinforcement learning on Muse Spark Open, using vLLM for rollout and verl FSDP2 for training.
The NVFP4 base serves rollouts, a BF16 dequantization of the same weights is used for training, and only the LoRA adapter is updated and synced back.
With GRPO on DAPO-Math-17K over 150 steps on 16 GB300 GPUs, AIME-2024 mean@8 rose from 0.56 to 0.85, with train-rollout k3 KL near 2 × 10⁻⁴.
Part of a thread · earlier post
Day-0 LoRA RL is supported on Muse Spark Open: vLLM for rollout, verl FSDP2 for training.
vLLM serves the NVFP4 base with LoRA; the trainer uses a BF16 dequantization of the same weights. R3 replays rollout routing, and only the adapter is updated and synced back.
With GRPO on DAPO-Math-17K, 150 steps on 16 GB300 GPUs improved AIME-2024 mean@8 from 0.56 to 0.85. Train–rollout k3 KL stayed near 2 × 10⁻⁴.
Titan RL also uses vLLM as its rollout engine to provide day-0 RL support.
5/6
Source: vLLM · x.comPublished · added here
