Vasanthi Jagatha + Somnath Roy (@SamsungUS) on memory-accelerated prefix serving for HiCache.
Their proposed lightweight hot/cold hints guide KV placement, promotion and demotion across memory tiers without changing the existing API.
Samsung researchers propose hot/cold hints for HiCache KV placement
AIVasanthi Jagatha and Somnath Roy of Samsung propose lightweight hot/cold hints that guide KV placement, promotion and demotion across memory tiers for memory-accelerated prefix serving in HiCache. The hints work without changing the existing API.
Source details
LMSYS Org · x.comPublished
Original title: Vasanthi Jagatha + Somnath Roy (@SamsungUS) on memory-accelerated prefix serving for HiCache.