Salesforce paper compares distillation and RL training compute, finds 42% savings from expert merging

A Salesforce paper compares on-policy distillation with GRPO, a reinforcement learning method, by held-out accuracy per training FLOP. According to a summary by Rohan Paul on X, with an external post-trained teacher, GRPO generally won, and distillation often inflated response length.

The paper also reports that distilling per-domain GRPO experts into one student matched joint GRPO on math plus code with 42.0% fewer total FLOPs. Another summary of the paper says distilling from a large external teacher usually costs more compute than plain RL.