Modal says off-policy correction beats full sampler-trainer alignment in RL
Article timeline
The articles in this story. Times are ET.
Modal@modalOfficialModal argues off-policy correction matters more than full RL sampler alignmentAIModal says fully aligning the sampler with the trainer for on-policy reinforcement learning is expensive, and partially aligning only some aspects still leaves the sampler-trainer mismatch growing. The post argues that off-policy correction, which fixes this growth, matters more.
- Modal@modalOfficialModal reports a heterogeneous sampler-trainer setup that is 15% cheaper
AIModal says a heterogeneous sampler-trainer configuration matches the performance of homogeneous samplers while costing 15% less, despite a 4.7× larger sampler-trainer KL on average. The company notes the configuration was not heavily optimized for cost, so further savings are available.
- Modal@modalOfficialModal trains agentic coding model with mixed-accelerator RL samplers
AIModal says it trained a model on agentic coding tasks using fully asynchronous reinforcement learning with mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at lower cost. The approach lets training use inference optimizations across vendors and hardware generations rather than a single fixed cluster of the latest accelerators.
- Modal@modalOfficialModal runs RL with mixed-precision samplers on B200 GPUs
AIModal uses Miles from RadixArk as its RL framework, with SGLang samplers and a Megatron-LM trainer that updates the policy in BF16 on B200 GPUs. It runs two setups: mixed samplers spanning six accelerator types and three precisions (BF16, FP8 and NVFP4), and homogeneous samplers that run only on B200s with BF16 weights and a BF16 KV cache.
