vLLM· vllm_project· Published Oct 7, 4:42 PM ET· added Oct 8, 8:55 PM ET· 2d ago28/100AI score28/100

vLLM bounded SWA replay cuts prefill compute 30–40% after prefix hits

Original title2/ SWA bounded replay: rebuilding sliding-window KV exactly after a prefix hit means replaying 40 × 128 tokens. Bounded replay reruns onl...

AISummary

vLLM's bounded sliding-window attention (SWA) replay reruns only each request's last 128 tokens after a prefix hit, instead of replaying 40 × 128 tokens. With CUDA graphs, prefill compute drops 30–40% for layers 21–39.

Read the original x.com

Source: vLLM · x.comPublished Oct 7, 4:42 PM ET · added here Oct 8, 8:55 PM ET

Open-source ecosystem Deployment & engineering Tutorials & how-to #Tutorial/How-to #Open source/Repo