Moore Threads open-sources Megatron-MUSA-Titan for trillion-parameter MoE training on its GPUs
AIMoore Threads has open-sourced Megatron-MUSA-Titan, training infrastructure for trillion-parameter mixture-of-experts models on clusters of more than 10,000 of its GPUs. The company says the stack has been used for trillion-parameter MoE pretraining, but has not published a specific MFU figure.
Source details
Pandaily · contact@pandaily.com (Pandaily) · pandaily.comPublished · added here