Perplexity and NVIDIA publish JOLT, a self-distillation and RL post-training method

AIAravind Srinivas shares Perplexity research on JOLT, a post-training method in which one model acts as teacher and student. The teacher is trained on its own attempts with rewards and penalized for drifting from the student, and the authors report that JOLT matches GRPO on MATH with 6.5x fewer training tokens.

AIWhy it matters.

The chart compares JOLT and GRPO on processed training tokens, showing where each reaches the same MATH accuracy and how much compute that difference represents.

Aravind SrinivasOfficial

@AravSrinivas

View on X

Sharing new post-training research on self-distillation + RL (JOLT), done together with @NVIDIAAI.

In self-distillation, the same model performs the role of both a teacher and a student. The teacher gets extra information that the student doesn't, and the student learns from feedback provided by the teacher. However, the teacher could still use shortcuts in ways the student has no way to follow.

Our proposed approach, JOLT, trains the teacher as well, to learn from rewards on its own attempts, while penalizing any drift that's far off from what the student model would do. JOLT+ further supplements JOLT by providing the student model outcome-based rewards in addition to the teacher supervision signals.

On the MATH benchmark, we found that JOLT matches GRPO with 6.5x fewer training tokens, while JOLT+ convincingly beats GRPO across math, coding, terminal tasks, and tool use benchmarks.

Image from @perplexity_ai's post — Perplexity@perplexity_ai

Source details

Aravind Srinivas · x.comPublished Oct 21, 5:04 PM ET · added here 5:10 PM ET

Original title: Sharing new post-training research on self-distillation + RL (JOLT), done together with @NVIDIAAI.

Perplexity and NVIDIA publish JOLT, a self-distillation and RL post-training method

AIWhy it matters.

Topics