Skip to content
View original post on X: vLLMOfficial· 22/100AI score22/100

vLLM cuts sparse prefill indexer and top-k kernel latency on B300

AISummary

vLLM reports that its sparse prefill path now runs its indexer logits kernel 43% faster and its top-k kernel 84% faster, cutting overall prefill latency by 2.2%. The measurements come from B300 DEP8 runs with MTP3, FP8 KV, a 60K-token input and the MegaMoE path. The change also removes 7 runtime JIT warmups and caps chunks at 4,096 tokens.

Post on XView on X
vLLMVerified on X
@vllm_project

Part of a thread · earlier post

Less work in the sparse prefill path:

  • Indexer logits kernel: 43% lower latency
  • Top-k kernel: 84% lower latency
  • Overall prefill: 2.2% lower latency

These are B300 DEP8 measurements with MTP3, FP8 KV, a 60K-token input and the MegaMoE path. The work also removes 7 runtime JIT warmups and caps chunks at 4,096 tokens.

3/6

Source: vLLM · x.comPublished

Topics

Open-source ecosystemDeployment & engineeringProduct updates