AWS walkthrough: multi-node disaggregated LLM inference with Ray LLM DLC on Amazon EKS
Overview
AWS Machine Learning Blog publishes a walkthrough of deploying one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Containers.
The setup places prefill and decode on separate nodes behind one OpenAI-compatible endpoint, with KV-cache transfer over NIXL and Elastic Fabric Adapter. The post describes a Qwen model deployment.
The complete code is available in an accompanying repository. The deployment requires service quota for two g6.8xlarge instances, at least 64 vCPU, in the target Region.
Written by AI from the articles below · updated Oct 12, 12:51 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- AWS Machine Learning BlogOfficialAWS shows how to run multi-node inference with the Ray LLM Deep Learning Containers on EKS
AIAWS explains how to deploy one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Containers. The setup separates prefill and decode onto different nodes behind one OpenAI-compatible endpoint, with KV-cache transfer over NIXL and Elastic Fabric Adapter.
Heat trend
Not enough continuous observations to show a trend yet.