SGLang adds unified memory so GPU capacity follows demand for hybrid models

AISGLang introduces Unified Memory, a shared arena that allocates GPU memory dynamically across full-attention KV, sliding-window KV, and recurrent state. LMSYS Org says fixed memory partitions waste GPU capacity in hybrid model inference, and the arena removes partitioning knobs, supports any page size, and includes speculative draft KV. The image shows fixed partitions stranding capacity compared with a single shared arena where capacity follows demand.

AIWhy it matters.

The post explains how fixed memory partitions waste GPU capacity for hybrid models, a concrete problem that makes the shared arena design worth understanding.

LMSYS OrgOfficial

@lmsysorg

View on X

🚀 Unified Memory in SGLang: One Arena for Every Token Lifetime

Hybrid models mix full-attention KV, sliding-window KV, and recurrent state. Fixed memory partitions waste GPU capacity. We built a shared arena where memory dynamically follows demand.

Key innovations:
🔹 No partitioning knobs, any page size, zero cross-type fragmentation
🔹 Token-centric KV layout, including speculative draft KV
🔹 Smart admission that rearranges memory to fit new requests
🔹 Safe asynchronous page reuse during in-flight GPU work

Read the full blog 👇

Source details

LMSYS Org · x.comPublished Nov 11, 6:05 PM ET · added here 6:14 PM ET

Original title: 🚀 Unified Memory in SGLang: One Arena for Every Token Lifetime

SGLang adds unified memory so GPU capacity follows demand for hybrid models

AIWhy it matters.

Topics