Great work from @zwljohnny, @tri_dao, and @NVIDIA on ReplaySSM!
Mamba2 standard decode and FlashInfer-backed MTP support are already in vLLM, reducing recurrent-state memory traffic by caching SSM inputs and updating the checkpoint only when needed.
vLLM shares ReplaySSM, a decoding method that caches SSM inputs
AISummary
vLLM, with NVIDIA and Tri Dao, released a paper, blog post, and code for ReplaySSM, a method that caches SSM inputs instead of the state.
The method separates output computation from state updates during decoding, which the authors say yields speedups of up to 1.47x for standard decoding and up to 2.1x for speculative decoding.
It also enables a low-precision SSM state whose rounding error does not propagate through the recurrence, and it ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang.
AIWhy it matters
The post explains a decoding change that separates output computation from state updates, showing where the reported speedups and low-precision gains come from.
Post on XView on X
vLLMVerified on X
@vllm_project
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang! It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence. Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation. Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
Source: vLLM · x.comPublished · added here