Moore Threads open-sources Megatron-MUSA-Titan for trillion-parameter MoE training
Overview
Moore Threads has open-sourced Megatron-MUSA-Titan, training infrastructure for trillion-parameter mixture-of-experts models on clusters of more than 10,000 of its GPUs, according to Pandaily.
The company says the stack has been used for trillion-parameter MoE pretraining but has not published a specific MFU figure. The code is on GitHub.
The project is built on NVIDIA's Megatron-LM. Moore Threads says it runs on its GPUs through torch_musa and MUSA C without modifying upstream code, and that it supports blockwise FP8 training and reinforcement learning.
Written by AI from the articles below · updated Oct 28, 10:12 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- PandailyNewsMoore Threads open-sources Megatron-MUSA-Titan for trillion-parameter MoE training on its GPUs
AIMoore Threads has open-sourced Megatron-MUSA-Titan, training infrastructure for trillion-parameter mixture-of-experts models on clusters of more than 10,000 of its GPUs. The company says the stack has been used for trillion-parameter MoE pretraining, but has not published a specific MFU figure. The project is built on NVIDIA's Megatron-LM, and the company says it runs on its GPUs through torch_musa and MUSA C without modifying upstream code, with blockwise FP8 training and reinforcement learning support.
Heat Trend
Not enough continuous observations to show a trend yet.