2 | @Cloudflare
Every Spare Gigabyte: A Fleet-Wide Elastic Mooncake Cache
Kastan Day on turning spare RAM and SSD into a shared KV cache that grows and shrinks with model demand.
Cloudflare explains a fleet-wide elastic Mooncake KV cache
AICloudflare explains how it turns spare RAM and SSD capacity into a shared KV cache that grows and shrinks with model demand. Kastan Day presents the approach as a way to use otherwise idle gigabytes across its fleet.
@lmsysorg
Part of a thread · earlier post