SemiAnalysis explains DRAM KVCache offloading to raise cached tokens for agentic inference

SemiAnalysis says that offloading KVCache from GPU memory to CPU DRAM raises cacheable tokens by 20-40% at 12-16 concurrent agentic sessions on a B200.

The post says a B200 has 180 GB of HBM, and after model weights only 100 GB per chip remains for KVCache, which it calls insufficient for long-context agentic inference workloads.

The post names SGLang HiCache as one way to do the offloading, moving KVCache into CPU DDR5 or LPDDR5X memory.