# Hugging Face publishes a free Ultimate MoE Training Handbook from 800k GPU hours of experiments

> Original title: Hugging Face 团队投入 ~80 万 GPU 小时开展实验，整理成免费开放的「终极 MoE 训练手册」

- Source: meng shao
- Published: 2026-11-06T12:21:47.000Z
- Bad Influence: https://badinfluence.news/items/uu97kgcdpzrhw81t9pr33bscm
- Original: https://x.com/shao__meng/status/2118680242797883876

## Summary

Hugging Face says it ran about 800k GPU hours of MoE experiments and released the results as The Ultimate MoE Training Handbook, a free, open guide of about 70k words with 55 interactive figures and runnable code. The guide covers expert count and size, all-to-all communication overhead, router choice with FP32 precision, and auxiliary loss versus loss-free bias for load balancing. It also addresses upcycling a dense checkpoint versus training from scratch, and reducing active experts at serving time.

## Post

Hugging Face 团队投入 ~80 万 GPU 小时开展实验，整理成免费开放的「终极 MoE 训练手册」

全文约 7 万词，配有 55 个交互图示、可运行代码和实际实验数据，围绕训练团队最常遇到的问题展开：专家数量与大小怎么选、跨卡通信为何拖慢训练、路由与负载均衡如何设置，以及该从稠密模型改造还是从头训练。是一份把分散的 MoE 训练经验系统公开、供实践者反复查阅的工程指南。
https://huggingface.co/spaces/huggingface/moe-blogpost

MoE 到底省了什么

MoE 把部分前馈网络换成多个专家，由路由器为每个 token 选择其中几个，再合并它们的输出。因此，模型可以增加总参数量，而不按相同比例增加每个 token 的计算量。

文中示例有 64 个专家，每次选 8 个，每个专家的宽度为 1024。相较宽度 8192 的稠密前馈层，它存储约 8 倍参数，每个 token 使用的专家总宽度却相同。省的是激活计算，不是全部参数的存储需求；整个示例模型约有 68.6 亿总参数、12.2 亿激活参数。

真正难在三个环节

1. 路由要稳定。 专家选择依赖分数排序，低精度可能改变接近的分数与选择结果。手册因此强调路由计算采用 FP32，并讨论不同路由方法的取舍。

2. 专家负载要均衡。 如果大量 token 涌向少数专家，其他设备就会等待。文章比较辅助损失、无需辅助损失的偏置调整等方法。强制均匀路由的对照实验约快 1.5 倍，但它舍弃了学到的路由，只能说明速度空间，不能直接当成训练方案。

3. 通信不能抵消计算收益。 专家分布在不同 GPU 后，每层都要分发 token、收回结果。文中一次性能分析里，未被计算掩盖的通信约占一步的一半；网络配置错误、通信退回普通 socket 的案例甚至慢了约 25 倍。

从训练延伸到部署与强化学习

手册建议先检查硬件和网络、建立稠密模型基线，再逐项加入 MoE 设置，而不是一次打开所有优化。

部署时，减少每次选中的专家数可以降低计算量，但不会自动减少模型权重的显存占用，也可能影响质量。强化学习还有额外问题：生成训练数据的推理引擎，与训练引擎可能选出不同专家，造成执行路径不一致。

## AI translation

Hugging Face team spent ~800,000 GPU hours on experiments and compiled them into a free, open "Ultimate MoE Training Handbook"

The full text is about 70,000 words, with 55 interactive figures, runnable code, and real experimental data. It centers on the problems training teams most often run into: how to choose the number and size of experts, why cross-GPU communication slows training, how to set up routing and load balancing, and whether to convert a dense model or train from scratch. It is an engineering guide that systematically makes scattered MoE training experience public, for practitioners to consult repeatedly.
https://huggingface.co/spaces/huggingface/moe-blogpost

What exactly does MoE save?

MoE replaces part of the feed-forward network with multiple experts. A router selects several of them for each token, and then merges their outputs. As a result, a model can increase its total parameter count without increasing per-token computation in the same proportion.

The example in the text has 64 experts, selects 8 for each token, and gives each expert a width of 1024. Compared with a dense feed-forward layer of width 8192, it stores about 8 times as many parameters, yet the total expert width each token uses is the same. What it saves is activated computation, not the storage needed for all parameters; the example model as a whole has about 6.86 billion total parameters and 1.22 billion activated parameters.

The real difficulty lies in three areas

1. Routing must be stable. Expert selection depends on score ranking, and low precision can change close scores and therefore the selection results. The handbook therefore stresses computing routing in FP32, and discusses the trade-offs between different routing methods.

2. Expert load must be balanced. If large numbers of tokens flock to a few experts, other devices have to wait. The article compares methods such as auxiliary losses and bias adjustment that needs no auxiliary loss. A control experiment with forced uniform routing ran about 1.5 times faster, but it discards the learned routing, so it only shows the speed headroom and cannot be used directly as a training scheme.

3. Communication must not cancel out the computational gains. Once experts are spread across different GPUs, every layer has to dispatch tokens and collect results. In one performance analysis in the text, communication not hidden by computation accounted for about half of a step; in a case where a network misconfiguration made communication fall back to ordinary sockets, the run was even about 25 times slower.

Extending from training to deployment and reinforcement learning

The handbook recommends first checking hardware and networking and establishing a dense-model baseline, then adding MoE settings one at a time, rather than turning on all optimizations at once.

When deploying, reducing the number of experts selected per token lowers computation, but it does not automatically reduce the VRAM needed to hold the model weights, and it may affect quality. Reinforcement learning raises an additional problem: the inference engine that generates training data and the training engine may select different experts, causing their execution paths to diverge.

[Video](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Famplify_video_thumb%2F2118680174917201922%2Fimg%2FeqEqm1ITy_8C2vdQ.jpg&mode=thumb&exp=1794182400&sig=0036c1be2d53d073)

## Quoted post @Thom_Wolf

> 18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
> 
> And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
> 
> So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
> - how many experts, how big, shared or not?
> - why is half my step spent in all-to-all, and how do I hide it?
> - why is my run 25x slower with no error in the logs?
> - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
> - aux loss or DeepSeek's loss-free bias?
> - upcycle a dense checkpoint or train from scratch?
> - can I serve with fewer experts per token than I trained with?
> 
> ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
> 
> Read it here: https://huggingface.co/spaces/huggingface/moe-blogpost

https://x.com/Thom_Wolf/status/2118432475730882627