Buoyant launches Kinference, an open-source inference control plane for vLLM
AIBuoyant has released Kinference, an open-source Apache v2 inference control plane for vLLM and Kubernetes, available on GitHub. It adds token rate limiting, per-tenant usage and cost tracking, KV-cache-aware load balancing, and YAML inference policies for routing or bursting requests.
AIWhy it matters.
The post benchmarks cache-aware routing against standard Kubernetes load balancing, with TTFT and goodput figures that show how routing choices affect serving performance.
Source details
vLLM Blog · the Kinference Team · vllm.aiPublished · added here
Original title: Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes