AWS details push-based NIXL KV transfer for disaggregated vLLM inference
Overview
AWS Machine Learning Blog introduces a push-based NIXL connector for vLLM that moves KV cache transfer setup off the time-to-first-token critical path.
In a 1P1D benchmark on Amazon EC2 P5en instances with EFA running Qwen3-32B, AWS reports up to 3x better TTFT and up to 30 percent better TPOT than the default pull mode.
To use push mode, both the prefill and decode servers must be configured with NixlPushConnector instead of NixlConnector. vLLM supports both push- and pull-based KV connectors, so users can choose the one that fits their deployment.
Written by AI from the articles below · updated Oct 19, 12:31 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialPush-based KV transfer cuts vLLM disaggregated inference TTFT up to 3x
AIAWS Machine Learning Blog introduces a push-based NIXL connector for vLLM that moves KV cache transfer setup off the time-to-first-token critical path. In a 1P1D benchmark on Amazon EC2 P5en instances with EFA running Qwen3-32B, push mode delivered up to 3x better TTFT and up to 30 percent better TPOT than the default pull mode.
Heat Trend
Not enough continuous observations to show a trend yet.