llm-d 1.0 inference framework released with multi-tenant QoS and zero-downtime operations
Overview
IBM, Red Hat and Google, the lead maintainers of the open-source llm-d project, release version 1.0 of its Kubernetes-based distributed AI inference framework.
The release adds quality-of-service controls across multiple tenants, zero-downtime upgrades, and routing that reuses cached context for AI agents. The framework is currently hosted as a sandbox project by the Cloud Native Computing Foundation.
IBM says a test of 3,000 concurrent software-engineering agents on self-hosted infrastructure cost five to 10 times less than a hosted service serving the same open model. The source does not give the test's hardware or the hosted service's name.
Written by AI from the articles below · updated Oct 20, 8:58 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- SiliconANGLE · AINewsIBM-backed llm-d releases 1.0 with controls for running AI models in-house
AIIBM, Red Hat and Google, the lead maintainers of the open-source llm-d project, release version 1.0 of its Kubernetes-based distributed inference framework. The release adds multi-tenant quality-of-service controls, zero-downtime upgrades and routing that reuses cached context for AI agents. IBM says a test of 3,000 concurrent software-engineering agents on self-hosted infrastructure cost five to 10 times less than a hosted service serving the same open model.
Heat Trend
Not enough continuous observations to show a trend yet.