Hugging Face publishes free MoE training handbook built on GPU experiments
Article timeline
The articles in this story. Times are ET.
Thomas WolfOfficialNouamane shares a guide to large-scale MoE trainingAINouamane, following his Ultra Scale Playbook, has published a guide to training mixture-of-experts models at large scale. Leandro Von Werra calls it probably the most comprehensive resource on the topic, featuring extensive visualizations.
- Thomas WolfOfficialPickHugging Face publishes a handbook on MoE training optimization
AIHugging Face has published a MoE training handbook, which it says covers everything from profiling to large cluster training. The team says it spent 1/2M GPUh optimizing MoE training, and the blog runs about 30k words with 50+ interactive figures.
- Thomas WolfOfficialHugging Face releases a free MoE training handbook based on 800k GPU hours of experiments
AIHugging Face says it ran 800k GPU hours of mixture-of-experts experiments and released the results as The Ultimate MoE Training Handbook, free and open. The handbook covers questions such as expert count and size, all-to-all communication overhead, router choice, and whether to upcycle a dense checkpoint. It contains about 70k words, 55 interactive figures, and runnable code.
Leandro von WerraOfficialHugging Face publishes a guide to training mixture-of-experts modelsAIHugging Face publishes a 30,000-word guide to training mixture-of-experts models, based on 500,000 GPU hours spent optimizing MoE training. The guide has more than 50 interactive figures and covers topics from profiling to large cluster training. Leandro von Werra shares the post and Nouamane Tazi's description of it as a comprehensive resource.