Lewis Tunstall describes SFT and RL training without a stronger teacher model

Lewis Tunstall of Hugging Face says distillation, which trains a model on outputs from a stronger model, does not work when no stronger model exists.

In an X thread, he says his team ran into this problem while building the SFT and RL subsets of FineDNA and describes the approaches that worked for them.