Leandro's thread:
https://x.com/lvwerra/status/2118415483603177956?s=20
Nouamane shares a guide to large-scale MoE training
AINouamane, following his Ultra Scale Playbook, has published a guide to training mixture-of-experts models at large scale. Leandro Von Werra calls it probably the most comprehensive resource on the topic, featuring extensive visualizations.
@Thom_Wolf
Part of a thread · earlier post
After the Ultra Scale Playbook, Nouamane is back with this crazy guide to train MoEs! It's probably the most comprehensive resource if you want to get into large scale MoE training and an absolute gem packed with beautiful visualization!View quoted post on X