For local serving, MXFP4 experts and block-FP8 dense layers cut weights from ~534GB to ~292GB; a tested Hopper setup moved from 8 GPUs to 4. In separate tests, 8 GPUs averaged 604ms to first token and 4 GPUs reached ~1.36 QPS. Workloads differed.
Ant Ling cuts local model weights from 534GB to 292GB with MXFP4 and FP8
AISummary
Ant Ling says MXFP4 experts and block-FP8 dense layers reduce weights from about 534GB to about 292GB for local serving. A tested Hopper setup moved from 8 GPUs to 4. In separate tests, 8 GPUs averaged 604ms to first token, and 4 GPUs reached about 1.36 QPS, though the workloads differed.
Post on XView on X
Source: Ant Ling · x.comPublished · added here