vLLM Elastic EP scaling cuts DeepSeek V4 Pro downtime to 0.82 seconds
Overview
vLLM says its Elastic EP scaling now reduces serving downtime to 0.82 seconds when DeepSeek V4 Pro scales from 16 to 20 GPUs.
The vLLM Blog attributes the gain to background preparation of joining workers, NIXL EP's in-place rank changes, and CUDA graphs kept on existing workers. The same benchmark took 1.96 seconds to scale back from 20 to 16 GPUs, according to the post.
Itay Alroy of NVIDIA describes the near-zero-downtime approach, which combines asynchronous preparation with NIXL EP. He says background preparation, CUDA graph reuse, and independent graph capture on joining workers are available in the latest vLLM and NIXL releases, installable with pip.
Written by AI from the articles below · updated Oct 19, 5:37 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- vLLM BlogOfficialvLLM cuts Elastic EP scaling downtime to under two seconds on DeepSeek V4 Pro
AIvLLM's Elastic EP scaling now reduces serving downtime to 0.82 seconds when scaling DeepSeek V4 Pro from 16 to 20 GPUs. The source attributes this to background preparation of joining workers, NIXL EP's in-place rank changes, and preserved CUDA graphs on existing workers. The authors say the same benchmark took 1.96 seconds to scale back from 20 to 16 GPUs.
Heat Trend
Not enough continuous observations to show a trend yet.