Skip to content
Trending storyDeveloping

vLLM tests prefill-decode split on 12 GB300 GPUs in pre-release prototype

1 article1 sourcesince Oct 14Last article 1h ago ·

Overview

AISummary of 1 article

vLLM reports that its prefill-decode (P/D) prototype ran on 12 GB300 GPUs, connecting DEP4 prefill to DEP8 decode with FP8, NIXL and a 384 GiB cache store.

The test recorded 514,784 prefix-hit tokens (91.24%), and two 1,000-request OCR runs scored 90.4% and 89.4% with zero request errors, according to vLLM's post on X.

The test ran before release and rebase, and the P/D pull request remains open, so the final build may differ.

Written by AI from the articles below · updated Oct 14, 11:29 PM ET

Check the sources:

Article timeline

The articles in this story. Times are ET.

Oct 14
  1. vLLMOfficial
    vLLM prototypes prefill-decode split on 12 GB300 GPUs with FP8

    AIvLLM's prototype connects DEP4 prefill to DEP8 decode on 12 GB300 GPUs using FP8, NIXL and a 384 GiB cache store. The test recorded 514,784 prefix-hit tokens (91.24%) and two 1,000-request OCR runs scored 90.4% and 89.4% with zero request errors. The test ran before release and rebase, and the P/D pull request remains open.

    Image from @vllm_project's post

Heat trend

Not enough continuous observations to show a trend yet.