Skip to content
View original post on X: SemiAnalysisX· 50/100AI score50/100

AMD lands megakernel decode support in upstream vLLM before NVIDIA

AISummary

AMD has landed megakernel decode support in upstream vLLM, a feature available on ROCm but not yet on CUDA, according to SemiAnalysis. The technique reduces kernel launches by 10x to improve high-interactivity decoding, and CUDA vLLM has no draft PRs for upstream megakernel support yet.

Post on XView on X
SemiAnalysisVerified on X
@SemiAnalysis_

ALERT🚨: AMD JUST LANDED SUPPORT FOR MEGAKERNEL DECODE IN UPSTREAM VLLM WELL BEFORE NVIDIA. This marks one of the first times that a major vLLM feature is only supported on ROCm and not on CUDA yet.

MegaKernels is a technique that is massively important for high-interactivity decoding by reducing the # of kernel launches by 10x.

This marks an❗️important milestone❗️for ROCm vLLM to land major ROCm-exclusive features. CUDA vLLM doesn't even have draft PRs for upstream vLLM megakernel support yet.

Source: SemiAnalysis · x.comPublished · added here

Topics

Open-source ecosystemDeployment & engineeringProduct updates