Zyphra reports MoE token-exchange overhead of 13-24% of step time on one node and 45-60% across four nodes
Overview
In Zyphra's runs, MoE token exchange between experts consumed 13-24% of step time on one node and 45-60% across four nodes.
Because experts are spread across GPUs and nodes, tokens must be sent to their experts and returned, making this communication a major training cost as models scale.
Written by AI from one article, by Zyphra
Check the sources:
Article timeline
Follow the coverage from different perspectives. Times are ET.
- ZyphraMoE training spends 45-60% of step time on cross-node token exchange
AIIn Zyphra's runs, MoE token exchange between experts consumed 13-24% of step time on one node and 45-60% across four nodes. Because experts are spread across GPUs and nodes, tokens must be sent to their experts and returned, making this communication a major training cost as models scale.
Heat trend
Not enough continuous observations to show a trend yet.