NVIDIA explains five system-level methods to optimize AI inference memory

AINVIDIA says it reduces AI inference memory pressure by codesigning GPUs, CPUs, networking, storage and software, rather than simply adding more HBM per GPU. Its techniques include NVFP4 weight and KV-cache formats, distributing weights across up to 72 GPUs in an NVL72 system, and offloading inactive context to CMX context memory storage through NVIDIA Dynamo. NVIDIA reports that HBM capacity per GPU rose about 3% from Blackwell Ultra to Vera Rubin while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.

AINVIDIA's technical blog says every byte of high bandwidth memory in an AI factory should produce tokens, not hold redundant weight copies, idle session KV caches, or recomputed prefixes. The post presents five ways NVIDIA co-design aims to maximize the useful work each byte delivers.