Epoch AI introduces an Inference Performance Index for comparing chip capacity
AIEpoch AI introduces the Inference Performance Index (IPI), which scores chips by how many agentic requests they can serve at once across output speeds of 50 to 400 tokens per second per request. Epoch estimates GB300 offers about 25 times H100's inference capacity per chip on the modeled workloads, despite only 2.5 times the 8-bit FLOP/s, because memory and inter-chip communication matter more than arithmetic throughput.
AIWhy it matters.
The report explains how memory and interconnect limits shape inference capacity, making chip comparisons based on raw FLOP/s less reliable.
Loading source posts…
Source details
Epoch AI · Jason Li · epoch.aiPublished · added here
Original title: Inference Performance Index: changing the way we measure inference capacity