Push-based KV transfer cuts vLLM disaggregated inference TTFT up to 3x
Original titlePush-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
AISummary
AWS Machine Learning Blog introduces a push-based NIXL connector for vLLM that moves KV cache transfer setup off the time-to-first-token critical path. In a 1P1D benchmark on Amazon EC2 P5en instances with EFA running Qwen3-32B, push mode delivered up to 3x better TTFT and up to 30 percent better TPOT than the default pull mode.
Source: AWS Machine Learning Blog · aws.amazon.comPublished · added here