SGLang splits sparse attention memory across 8 GPUs with no accuracy loss

AISGLang says it splits sparse attention's full-history memory across 8 GPUs using decode context parallelism while still selecting the same 2,048 tokens. The post says this allows up to 8x more room for history with no accuracy loss.

SGLangOfficial

@sgl_project

View on X

Part of a thread · earlier post

Sparse attention only reads the 2,048 most relevant tokens, but it still has to keep the whole history in memory.

We split that memory across 8 GPUs with decode context parallelism, and still find the exact same 2,048 tokens. That's up to 8x more room for the history, with no accuracy loss.

Source details

SGLang · x.comPublished Nov 4, 11:26 AM ET

Original title: Sparse attention only reads the 2,048 most relevant tokens, but it still has to keep the whole history in memory.

Topics

Deployment & engineering Tutorials & how-to