Skip to content
Trending storyDeveloping

SGLang adds Unified Memory shared GPU arena for hybrid model inference

1 article1 sourcesince Nov 11Last article

Overview

AISummary of 1 article

SGLang introduces Unified Memory, a shared arena that allocates GPU memory dynamically across full-attention KV cache, sliding-window KV cache, and recurrent state.

LMSYS Org says fixed memory partitions waste GPU capacity in hybrid model inference, and the arena removes the partitioning settings, supports any page size, and includes speculative draft KV.

The announcement is a single post on X from LMSYS Org, which includes an image comparing fixed partitions that strand capacity with one shared arena where capacity follows demand. The post does not report any measured throughput or memory-savings figures.

Written by AI from the articles below · updated Nov 11, 6:15 PM ET

Check the sources:

Latest developmentNov 11, 6:05 PM ET
SGLang adds unified memory so GPU capacity follows demand for hybrid models

Article timeline

The articles in this story. Times are ET.

Nov 11
  1. LMSYS OrgOfficial
    SGLang adds unified memory so GPU capacity follows demand for hybrid models

    AISGLang introduces Unified Memory, a shared arena that allocates GPU memory dynamically across full-attention KV, sliding-window KV, and recurrent state. LMSYS Org says fixed memory partitions waste GPU capacity in hybrid model inference, and the arena removes partitioning knobs, supports any page size, and includes speculative draft KV. The image shows fixed partitions stranding capacity compared with a single shared arena where capacity follows demand.

    Image from @lmsysorg's post

Heat Trend

Current comparable heat
7
Peak comparable heat
10Nov 11, 7:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.