Skip to content
Trending storyDeveloping

Cloudflare turns spare RAM and SSD into a shared KV cache for AI models

2 articles1 sourcesince Nov 4Last article 1h ago ·

Overview

AISummary of 2 articles

Cloudflare uses spare RAM and SSD capacity across its fleet as a shared KV cache for models, according to Kastan Day, who presented the approach.

Per Day, the cache grows and shrinks with model demand, so otherwise idle memory is put to use.

LMSYS Org, which shared Day's presentation on X, describes the design as a fleet-wide elastic Mooncake cache. The reports do not give capacity figures or performance results.

Written by AI from the articles below · updated Nov 4, 5:54 PM ET

Check the sources:

Latest developmentNov 4, 5:51 PM ET
Cloudflare turns spare RAM and SSD into a shared KV cache for models

Article timeline

The articles in this story. Times are ET.

Nov 4
  1. LMSYS OrgOfficial
    Cloudflare turns spare RAM and SSD into a shared KV cache for models

    AICloudflare's Kastan Day describes a fleet-wide elastic Mooncake cache that uses spare RAM and SSD as a shared KV cache. The cache grows and shrinks with model demand.

    Image from @lmsysorg's post
  2. LMSYS OrgOfficial
    Cloudflare explains a fleet-wide elastic Mooncake KV cache

    AICloudflare explains how it turns spare RAM and SSD capacity into a shared KV cache that grows and shrinks with model demand. Kastan Day presents the approach as a way to use otherwise idle gigabytes across its fleet.

    Image from @lmsysorg's post

Heat Trend

Not enough continuous observations to show a trend yet.