Skip to content
View original post on X: vLLMOfficial· 25/100AI score25/100

vLLM shares GB200 forward microbenchmark gains from Inferact's kernel work

AISummary

vLLM says Inferact's fused all-reduce plus norm, packed QKV paths, and tanh softcap kernel cut GB200 forward time from 4.23 to 3.89 ms at TP4 batch 1, and from 8.15 to 6.47 ms at TP8 batch 128. The figures come from forward microbenchmarks with dummy weights, not end-to-end serving results.

Post on XView on X
vLLMVerified on X
@vllm_project

Part of a thread · earlier post

Long-context serving needs more than a model loader.

Inferact's work includes fused all-reduce + norm, packed QKV paths and a tanh softcap kernel. In GB200 forward microbenchmarks, TP4 batch 1 fell from 4.23 to 3.89 ms; TP8 batch 128 from 8.15 to 6.47 ms.

These are forward microbenchmarks with dummy weights, not end-to-end serving results. The slide separates kernel timings from forward timings.

3/6

Source: vLLM · x.comPublished · added here

Topics

Open-source ecosystemDeployment & engineeringProduct updates