Hugging Face team spent ~800,000 GPU hours on experiments and compiled them into a free, open "Ultimate MoE Training Handbook"
The full text is about 70,000 words, with 55 interactive figures, runnable code, and real experimental data. It centers on the problems training teams most often run into: how to choose the number and size of experts, why cross-GPU communication slows training, how to set up routing and load balancing, and whether to convert a dense model or train from scratch. It is an engineering guide that systematically makes scattered MoE training experience public, for practitioners to consult repeatedly.
https://huggingface.co/spaces/huggingface/moe-blogpost
What exactly does MoE save?
MoE replaces part of the feed-forward network with multiple experts. A router selects several of them for each token, and then merges their outputs. As a result, a model can increase its total parameter count without increasing per-token computation in the same proportion.
The example in the text has 64 experts, selects 8 for each token, and gives each expert a width of 1024. Compared with a dense feed-forward layer of width 8192, it stores about 8 times as many parameters, yet the total expert width each token uses is the same. What it saves is activated computation, not the storage needed for all parameters; the example model as a whole has about 6.86 billion total parameters and 1.22 billion activated parameters.
The real difficulty lies in three areas
1. Routing must be stable. Expert selection depends on score ranking, and low precision can change close scores and therefore the selection results. The handbook therefore stresses computing routing in FP32, and discusses the trade-offs between different routing methods.
2. Expert load must be balanced. If large numbers of tokens flock to a few experts, other devices have to wait. The article compares methods such as auxiliary losses and bias adjustment that needs no auxiliary loss. A control experiment with forced uniform routing ran about 1.5 times faster, but it discards the learned routing, so it only shows the speed headroom and cannot be used directly as a training scheme.
3. Communication must not cancel out the computational gains. Once experts are spread across different GPUs, every layer has to dispatch tokens and collect results. In one performance analysis in the text, communication not hidden by computation accounted for about half of a step; in a case where a network misconfiguration made communication fall back to ordinary sockets, the run was even about 25 times slower.
Extending from training to deployment and reinforcement learning
The handbook recommends first checking hardware and networking and establishing a dense-model baseline, then adding MoE settings one at a time, rather than turning on all optimizations at once.
When deploying, reducing the number of experts selected per token lowers computation, but it does not automatically reduce the VRAM needed to hold the model weights, and it may affect quality. Reinforcement learning raises an additional problem: the inference engine that generates training data and the training engine may select different experts, causing their execution paths to diverge.