Zyphra says routing-pattern expert placement speeds MoE token communication up to 2.63x on AMD MI300X
Overview
Zyphra researchers report that exploiting patterns in how tokens are routed to experts makes Mixture of Experts communication up to 2.63x faster on AMD MI300X GPUs.
The speedup comes without changing the model itself.
Written by AI from one article, by Zyphra
Check the sources:
Developments
3 developments
- Oct 8, 11:35 AM ET · 1 articleZyphra proposes correlated expert placement to cut MoE token copies across GPUsZyphra: Zyphra's correlated expert placement cuts MoE token copies by 58%
- Oct 8, 11:35 AM ET · 1 articleZyphra reports expert-pair routing patterns in MoE layers and a method to exploit them for expert placementZyphra: Zyphra finds MoE expert routing has strong, exploitable pair patterns
- Oct 8, 11:35 AM ET · 1 articleZyphra uses token routing patterns to speed MoE expert communication up to 2.63x on AMD MI300XZyphra: Zyphra speeds MoE token routing up to 2.63x on AMD MI300X GPUs
Article timeline
Follow the coverage from different perspectives. Times are ET.
- ZyphraZyphra's correlated expert placement cuts MoE token copies by 58%
AIZyphra places experts that are frequently chosen together on the same GPU, so each token is sent to a GPU only once regardless of how many of its experts reside there. On 8 GPUs, this removes up to 58% of token copies sent across the network.
- ZyphraZyphra finds MoE expert routing has strong, exploitable pair patterns
AIIn one layer, just 64 of 8,128 possible expert pairs handled 42% of tokens, far above the 1.6% expected if experts were chosen independently. Zyphra says this structure can be exploited to improve expert placement.
- ZyphraZyphra speeds MoE token routing up to 2.63x on AMD MI300X GPUs
AIZyphra researchers report that exploiting patterns in how tokens are routed to experts makes Mixture of Experts communication up to 2.63x faster on AMD MI300X GPUs. The speedup comes without changing the model itself.
Heat trend
Not enough continuous observations to show a trend yet.