Skip to content
View original post on X: Lewis Tunstall
Published
14/100

Lewis Tunstall warns self-distillation on RLVR rollouts can cause hallucinations

AILewis Tunstall says self-distillation, which runs RLVR on a base model and fine-tunes on its correct rollouts, raises reward but can produce hallucinations. He says final-answer correctness rewards outputs such as citations to studies and databases that do not exist, so the training data needs closer inspection.
Lewis TunstallVerified on X
@_lewtun
View on X

Part of a thread · earlier post

The obvious alternative is self-distillation: run RLVR on the base model, keep the rollouts it gets right, and SFT on those.

Pros: reward goes up

Cons: final-answer correctness produces hallucinations like citing studies that don't exist, databases the works.

Conclusion? We're gonna need to look at the data 🙈

Source details

Lewis Tunstall · x.comPublished

Original title: The obvious alternative is self-distillation: run RLVR on the base model, keep the rollouts it gets right, and SFT on those.

Topics

ReasoningData & trainingSafety & alignmentExpert opinion