Skip to content
Trending storyDeveloping

Epoch AI introduces Inference Performance Index for comparing AI chip serving capacity

2 articles2 sourcessince Nov 8Last article 6h ago ·

Overview

AISummary of 2 articles

Epoch AI introduces the Inference Performance Index (IPI), a measure that scores chips by how many agentic requests they can serve at once across output speeds of 50 to 400 tokens per second per request.

Epoch says the index is needed because raw arithmetic throughput does not capture what matters for inference.

Using the IPI, Epoch estimates that Nvidia's GB300 offers about 25 times the inference capacity per chip of the H100 on the modeled workloads. That compares with a 2.5 times gain in 8-bit FLOP/s, which Epoch attributes to memory and inter-chip communication mattering more than arithmetic throughput. A later Epoch post on X says the index accounts for memory bandwidth and inter-chip communication alongside compute, and that FLOP/s alone understates GB300's advantage over H100 by 10 times. That post says IPI scores are expressed in GB300 equivalents and can be used to estimate a data center's total inference capacity.

Written by AI from the articles below · updated Nov 9, 4:18 PM ET

Check the sources:

Latest developmentNov 9, 2:06 PM ET
Epoch AI introduces an Inference Performance Index for AI chips

Article timeline

The articles in this story. Times are ET.

Nov 9
  1. Epoch AIOfficial
    Epoch AI introduces an Inference Performance Index for AI chips

    AIEpoch AI introduces the Inference Performance Index (IPI), a measure that accounts for memory bandwidth and inter-chip communication alongside compute. It says FLOP/s alone understates GB300's inference advantage over H100 by 10×. IPI scores are expressed in GB300 equivalents to compare chips directly and estimate a data center's total inference capacity.

    Horizontal bar chart ranking AI chips by modeled inference performance index normalized to Nvidia GB300=1, showing up to 72× higher inference capacity for newer chips than Nvidia H100.
Nov 8
  1. Epoch AIOfficialPick
    Epoch AI introduces an Inference Performance Index for comparing chip capacity

    AIEpoch AI introduces the Inference Performance Index (IPI), which scores chips by how many agentic requests they can serve at once across output speeds of 50 to 400 tokens per second per request. Epoch estimates GB300 offers about 25 times H100's inference capacity per chip on the modeled workloads, despite only 2.5 times the 8-bit FLOP/s, because memory and inter-chip communication matter more than arithmetic throughput.

Heat Trend

Current comparable heat
13
Peak comparable heat
15Nov 9, 3:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.