EUROPA explains how it trained a 1T-parameter MoE on JUPITER with PyTorch
AIThe EUROPA consortium says it raised training throughput on the JUPITER supercomputer from 90 to about 1,300 tokens/s per GPU across 384 GH200 GPUs. The post describes the changes behind the gain, including FlexShard placement for the Muon optimizer, activation-checkpoint offloading to the Grace CPU, and running FSDP's all-gather on the compute stream.
AIWhy it matters.
The post gives specific throughput changes and optimization steps for training a 1T-parameter MoE on shared public GPUs, useful for anyone facing similar interconnect or memory limits.
Source details
PyTorch Blog · The EUROPA consortium (Domyn, Fraunhofer IAIS and IIS) · pytorch.orgPublished · added here
Original title: Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan