Why not true on-policy RL? Full alignment of the sampler with the trainer is expensive. So why not align only certain aspects? Although the magnitude of the mismatch is lowered, it still continues to grow. Off-policy correction fixes this growth, which we argue matters more.
Modal argues off-policy correction matters more than full RL sampler alignment
AISummary
Modal says fully aligning the sampler with the trainer for on-policy reinforcement learning is expensive, and partially aligning only some aspects still leaves the sampler-trainer mismatch growing. The post argues that off-policy correction, which fixes this growth, matters more.
Post on XView on X
Source: Modal · x.comPublished · added here