Skip to content
View original post on X: LMSYS OrgOfficial· Pick62/100AI score62/100

LMSYS launches SGLang Low-Latency Mode with NVIDIA and AMD

AISummary

LMSYS Org announces SGLang Low-Latency Mode, built with NVIDIA and AMD. On DeepSeek-V4.1-Flash with DSpark, it raises P90 per-user token speed up to 1.38× on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in SemiAnalysis's AgentX trace replay. The post also says low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.

AIWhy it matters

The post gives per-GPU kernel and speed figures for NVIDIA and AMD hardware, showing how a low-latency inference mode changes serving performance for agent workloads.

Post on XView on X
LMSYS OrgVerified on X
@lmsysorg

Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD.

On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay.

Highlights:
- NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs
- AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them.
- New plugin system: custom kernels plug in from external libraries, with automatic fallback.
- Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.

Source: LMSYS Org · x.comPublished

Topics

NVIDIAOpen-source ecosystemDeployment & engineeringProduct updatesEvals & benchmarks