Samsung researchers propose hot/cold hints to guide HiCache KV placement across memory tiers

Vasanthi Jagatha and Somnath Roy of Samsung propose lightweight hot/cold hints for memory-accelerated prefix serving in HiCache, according to an X post from LMSYS Org.

The hints guide where KV cache is placed, promoted and demoted across memory tiers. The proposal works without changing the existing HiCache API. The report describes the approach only and gives no benchmark results.