# SGLang HiCache offloads KVCache to CPU memory for agentic inference

> Original title: ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concu...

- Source: SemiAnalysis
- Published: 2026-10-28T15:03:24.000Z
- Bad Influence: https://badinfluence.news/items/iknaaqrr23vc9qh4rekxgq4vq
- Original: https://x.com/SemiAnalysis_/status/2115459422696636806

## Summary

SemiAnalysis says enabling DRAM KVCache offloading raises cacheable tokens by 20-40% at 12-16 concurrent agentic sessions on a B200. The B200 has 180 GB of HBM, leaving only 100 GB per chip for KVCache after weights, which the post calls insufficient for long-context agentic inference. It says SGLang HiCache can also offload KVCache to CPU DDR5 or LPDDR5X memory.

## Post

ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concurrent agentic sessions. A B200 only has 180 GB of HBM, and after weight storage, only 100 GB of HBM per chip remains for KVCache.

THIS IS NOT ENOUGH FOR LONG-CONTEXT AGENTIC INFERENCE WORKLOADS. Thus, with SGLang HiCache, you can also offload KVCache to the CPU's DDR5/LPDDR5X memory, which helps improve performance. (1/2)🧵

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHVufInGWAAEXPvP.jpg&mode=thumb&exp=1793404800&sig=4dba17810d0c9513)