On-policy distillation provides an elegant way to use the teacher model as a process reward model to provide dense reward while preventing SFT style "OOD shock" during rollout.

Original titleOn-policy distillation provides an elegant way to use the teacher model as a process reward model to provide dense reward while preventin...

Source: Lilian Weng · x.com