Skip to content
View original post on X: SemiAnalysis
Published
23/100

SGLang HiCache offloads KVCache to CPU memory for agentic inference

AISemiAnalysis says enabling DRAM KVCache offloading raises cacheable tokens by 20-40% at 12-16 concurrent agentic sessions on a B200. The B200 has 180 GB of HBM, leaving only 100 GB per chip for KVCache after weights, which the post calls insufficient for long-context agentic inference.
SemiAnalysisVerified on X
@SemiAnalysis_
View on X

ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concurrent agentic sessions. A B200 only has 180 GB of HBM, and after weight storage, only 100 GB of HBM per chip remains for KVCache.

THIS IS NOT ENOUGH FOR LONG-CONTEXT AGENTIC INFERENCE WORKLOADS. Thus, with SGLang HiCache, you can also offload KVCache to the CPU's DDR5/LPDDR5X memory, which helps improve performance. (1/2)🧵

Source details

SemiAnalysis · x.comPublished

Original title: ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concu...

Topics

AI agentsDeployment & engineeringData & trainingTutorials & how-to