AWS contributes MoE kernel optimizations to vLLM for SageMaker AI inference
Overview
Amazon Web Services says it upstreamed Mixture-of-Experts kernel optimizations to vLLM, and that the changes are available in vLLM 0.24.0 or later.
AWS Machine Learning Blog describes a Swap-AB kernel change that reorders the MoE matrix multiplication to match Hopper WGMMA layouts, which it says improves end-to-end output throughput by 2–4 percent across large MoE models on H200 GPUs.
The post also says that for FP8 tensor-quantized models on Hopper, the Triton backend consistently outperforms the default FLASHINFER_CUTLASS path. AWS states it applies these optimizations in the Large Model Inference (LMI) v28 container, the Deep Learning Container on Amazon SageMaker AI.
Written by AI from the articles below · updated Oct 13, 12:14 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialSwap-AB kernel and Triton backend speed up MoE inference on Hopper GPUs
AIAWS Machine Learning Blog describes a Swap-AB kernel change that flips the MoE matrix multiplication to match Hopper WGMMA layouts. The change improves end-to-end output throughput by 2–4 percent across large MoE models on H200 and is available in vLLM 0.24.0 or later. For FP8 tensor-quantized models on Hopper, the post says the Triton backend consistently outperforms the default FLASHINFER_CUTLASS path.
Heat trend
Not enough continuous observations to show a trend yet.