Buoyant releases Kinference, an open-source inference control plane for vLLM
Overview
Buoyant has released Kinference, an open-source inference control plane for vLLM and Kubernetes, under the Apache v2 license and available on GitHub.
The tool adds token rate limiting, per-tenant usage and cost tracking, KV-cache-aware load balancing, and YAML inference policies for routing or bursting requests.
In Buoyant's benchmarks, Kinference cut median time to first token by 95% on one workload and delivered 7.4 times the goodput on a shared-prefix workload compared with standard Kubernetes load balancing. These results are Buoyant's own figures and have not been independently verified.
Written by AI from the articles below · updated Nov 4, 7:56 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- vLLM BlogOfficialBuoyant launches Kinference, an open-source inference control plane for vLLM
AIBuoyant has released Kinference, an open-source Apache v2 inference control plane for vLLM and Kubernetes, available on GitHub. It adds token rate limiting, per-tenant usage and cost tracking, KV-cache-aware load balancing, and YAML inference policies for routing or bursting requests. In Buoyant's benchmarks, Kinference cut median TTFT by 95% on one workload and served 7.4× the goodput on a shared-prefix workload compared with standard Kubernetes load balancing.
Heat Trend
Not enough continuous observations to show a trend yet.