2 | @Cloudflare
Every Spare Gigabyte: A Fleet-Wide Elastic Mooncake Cache
Kastan Day on turning spare RAM and SSD into a shared KV cache that grows and shrinks with model demand.
Cloudflare turns spare RAM and SSD into a shared KV cache for models
AICloudflare's Kastan Day describes a fleet-wide elastic Mooncake cache that uses spare RAM and SSD as a shared KV cache. The cache grows and shrinks with model demand.
@lmsysorg
Part of a thread · earlier post