After the Ultra Scale Playbook, Nouamane is back with this crazy guide to train MoEs!
It's probably the most comprehensive resource if you want to get into large scale MoE training and an absolute gem packed with beautiful visualization!
Hugging Face publishes a guide to training mixture-of-experts models
AIHugging Face publishes a 30,000-word guide to training mixture-of-experts models, based on 500,000 GPU hours spent optimizing MoE training. The guide has more than 50 interactive figures and covers topics from profiling to large cluster training.
@lvwerra
📖 The Ultimate MoE Training Handbook 📖 We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training. https://huggingface.co/spaces/huggingface/moe-blogpost
Source details
Leandro von Werra · x.comPublished · added here
Original title: After the Ultra Scale Playbook, Nouamane is back with this crazy guide to train MoEs!