vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads
Overview
vLLM's blog announces bi-directional KV transfer, which lets prefill nodes reuse KV blocks that decode nodes retained from earlier conversation turns, rather than recomputing the full history.
In AWS benchmarks on a 1P1D deployment using EC2 P5en instances, time to first token (TTFT) fell up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8 at long prompts. The blog says results depend on workload and configuration.
Using the feature requires vLLM v0.21.0 or later, NixlConnector, and a KV-cache-aware router.
Written by AI from the articles below · updated Oct 12, 2:55 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- vLLM BlogOfficialvLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads
AIvLLM's bi-directional KV transfer lets prefill nodes reuse KV blocks retained by decode nodes across conversation turns, instead of recomputing the full history. In AWS benchmarks on a 1P1D deployment with EC2 P5en instances, TTFT fell up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8 at long prompts. Results depend on workload and configuration, and the feature requires vLLM v0.21.0 or later, NixlConnector, and a KV-cache-aware router.
Heat trend
Not enough continuous observations to show a trend yet.