SSMs had the nice asymptotics but kept missing out on the inference goodies Transformers get for free like spec decode and quantization. Turns out we were just not caching the right thing. Same IO-aware trick behind FlashAttention, Mamba scan, paged KV. A lot of the big wins weren’t new architectures, just people taking memory IO seriously.
Tri Dao explains ReplaySSM, a caching change that speeds up SSM decoding
AISummary
Tri Dao says SSMs had good asymptotics but lacked inference tools like speculative decoding and quantization that Transformers get. He credits caching SSM inputs instead of the state, which separates output computation from state updates during decoding.
The quoted post says ReplaySSM runs in production in GLM-5.3-Flash and Kimi-K3 and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang, with decoding up to 1.47x faster and speculative decoding up to 2.1x faster.
Post on XView on X
Tri DaoVerified on X
@tri_dao
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang! It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence. Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation. Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
Source: Tri Dao · x.comPublished · added here