Skip to content
View original post on X: Lewis Tunstall
Published
22/100

Lewis Tunstall explains SFT and RL training without a stronger teacher model

AILewis Tunstall of Hugging Face says distillation, which trains a model on outputs from a stronger one, fails when no stronger model exists. He says his team faced this while building the SFT and RL subsets of FineDNA and describes the approaches that worked in a thread.
Lewis TunstallVerified on X
@_lewtun
View on X

Distillation is the most common trick in the post-training toolbox: sample from a stronger model, SFT on its outputs.

But what do you do when there's no stronger model to learn from?

We hit this problem while building the SFT and RL subsets of 🧬 FineDNA. Here's what worked 🧵

Source details

Lewis Tunstall · x.comPublished

Original title: Distillation is the most common trick in the post-training toolbox: sample from a stronger model, SFT on its outputs.

Topics

Hugging FaceData & trainingTutorials & how-to