SGLang announced a Low-Latency Mode built with NVIDIA and AMD that raises P90 per-user token speed on DeepSeek-V4.1-Flash.
In SemiAnalysis's AgentX trace replay, speed rose from 671 to 928 tok/s on NVIDIA GB300 (1.38x) and from 332 to 516 tok/s on AMD MI355X (1.56x).
SemiAnalysis said merged megakernel support brings GB300 close to 1000 tok/s per user, with low-latency kernels for GLM-5.3, K3, and MiniMax M3 in progress.
ALERT🚨: SGLANG JUST MERGED MEGAKERNEL SUPPORT, GETTING CLOSE TO 1000 tok/s/user on the GB300 NVL72 AgentX Harness & increasing interactivity by 1.5x on MI355X too! Great work by the SGLang, RadixArk, NVIDIA, and AMD teams!
Best of all, this already works out of the box on SGLang Docker images! Excited to rapidly get this on the official AgentX!
https://x.com/lmsysorg/status/2112342810376712527
Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD. On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay. Highlights: - NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs - AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them. - New plugin system: custom kernels plug in from external libraries, with automatic fallback. - Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.
Source: SemiAnalysis · x.comPublished · added here
