Skip to content
View original post on X: vLLMOfficial· 23/100AI score23/100

Inferact's kernel work cuts GB200 forward times in microbenchmarks

AISummary

Inferact's fused all-reduce plus norm, packed QKV, and tanh softcap kernels cut GB200 forward microbenchmark times, per vLLM. TP4 batch 1 fell from 4.23 ms to 3.89 ms, and TP8 batch 128 from 8.15 ms to 6.47 ms. The benchmarks used dummy weights and measured forward passes, not end-to-end serving.

Post on XView on X
vLLMVerified on X
@vllm_project

Part of a thread · earlier post

Long-context serving needs more than a model loader.

Inferact's work includes fused all-reduce + norm, packed QKV paths and a tanh softcap kernel. In GB200 forward microbenchmarks, TP4 batch 1 fell from 4.23 to 3.89 ms; TP8 batch 128 from 8.15 to 6.47 ms.

These are forward microbenchmarks with dummy weights, not end-to-end serving results. The slide separates kernel timings from forward timings.

Source: vLLM · x.comPublished · added here

Topics

Open-source ecosystemDeployment & engineeringProduct updates