NVIDIA explains full-stack codesign to ease AI inference memory pressure
AINVIDIA says it reduces AI inference memory pressure by codesigning GPUs, CPUs, networking, storage and software, rather than simply adding more HBM per GPU. Its techniques include NVFP4 weight and KV-cache formats, distributing weights across up to 72 GPUs in an NVL72 system, and offloading inactive context to CMX context memory storage through NVIDIA Dynamo. NVIDIA reports that HBM capacity per GPU rose about 3% from Blackwell Ultra to Vera Rubin while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.
Source details
NVIDIA Blog · Amr Elmeleegy · blogs.nvidia.comPublished · added here
Original title: More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge