# Hugging Face releases a free MoE training handbook based on 800k GPU hours of experiments

> Original title: 18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

- Source: Thomas Wolf
- Published: 2026-11-05T19:57:15.000Z
- Bad Influence: https://badinfluence.news/items/f9lmr17rhqbkztcqv18up6ydk
- Original: https://x.com/Thom_Wolf/status/2118432475730882627

## Summary

Hugging Face says it ran 800k GPU hours of mixture-of-experts experiments and released the results as The Ultimate MoE Training Handbook, free and open. The handbook covers questions such as expert count and size, all-to-all communication overhead, router choice, and whether to upcycle a dense checkpoint. It contains about 70k words, 55 interactive figures, and runnable code.

## Post

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://huggingface.co/spaces/huggingface/moe-blogpost

[Video](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Famplify_video_thumb%2F2118429152424370176%2Fimg%2FNrWnEgumFzh9jy5X.jpg&mode=thumb&exp=1794096000&sig=e68c901dca7d713f)