Skip to content
View original post on X: Sophia Yang
Published
59/100

ReplaySSM speeds up SSM decoding and runs in production in GLM-5.3-Flash and Kimi-K3

AIReplaySSM, a method for decoding state space models, is now in production in GLM-5.3-Flash and Kimi-K3 and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang. The method caches recent inputs instead of rewriting the full state every step, which speeds up standard decoding by up to 1.47x and speculative decoding by up to 2.1x.
Sophia YangVerified on X
@sophiamyang
View on X

ReplaySSM TLDR

- Problem: hybrid SSM models (Mamba-2, Gated DeltaNet) rewrite their full state to GPU memory on every decode step, and at serving batch sizes that memory traffic is the bottleneck.

- Fix: keep a checkpoint plus a buffer of recent inputs, and write the full state only occasionally.

- Result: decode runs 1.2–1.47x faster at batch size 256 across 4 hybrid models, with no gain at batch size 1. Speculative decoding gets about 1.8x vLLM's throughput on 120B-class models, because rolling back a wrong guess is just a pointer move.

Ze-Wei (Johnny) Liou@zwljohnny
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang! It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence. Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation. Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
View quoted post on X
Source details

Sophia Yang · x.comPublished · added here

Original title: ReplaySSM TLDR

Topics

NVIDIAOpen-source ecosystemDeployment & engineeringChina AIProduct updatesTutorials & how-to