Groq plans to stack DRAM in its next inference chip to get around SRAM’s small capacity, industry sources say.
Nvidia paid $20B last December to license Groq’s technology and hire its leaders, so this chip is effectively Nvidia’s next LPU.
The big deal here is that, each generated token requires reading the model’s weights from memory, so bandwidth sets speed.
The big deal here is that each generated token requires reading the model’s weights from memory, so bandwidth sets speed.
Today’s Groq 3 LPU reads its 500MB of on-chip SRAM at 150TB/s, about 7 times a Rubin GPU’s HBM bandwidth, which is fast enough for thousands of tokens per second.
That memory is so small, though, that a full 256-chip rack holds 128GB, less than one Rubin GPU’s 288GB, and an 8-bit trillion-parameter model needs roughly 2,000 chips just for its weights.
Stacking DRAM on the logic die can put far more memory beside the compute, and hybrid-bonded versions promise roughly 10 times HBM’s bandwidth density with 77% to 83% less energy per bit.
That would let fast inference on large models run on far fewer chips.
The reported design, known as 3DS, is more modest: it wires DRAM dies vertically through the silicon and keeps a standard DRAM interface, adding capacity mainly at ordinary DRAM speeds.
Groq hasn’t announced it, and earlier supply-chain reporting described another route: SK Hynix DRAM fused directly onto the logic die by hybrid bonding, alongside SRAM.
Cerebras plans to stack DRAM on its CS-6 wafer too, a sign the capacity ceiling is binding across SRAM-first designs.
So the bonding method decides the outcome: 3DS mostly adds capacity at DRAM speeds, while hybrid bonding promises bandwidth much closer to SRAM.