Skip to content
Trending storyDeveloping

Ant Ling cuts local model weights from 534GB to 292GB with MXFP4 and FP8

1 article1 sourcesince Oct 14Last article 1h ago ·

Overview

AISummary of 1 article

Ant Ling says it cut its model's weights from about 534GB to about 292GB for local serving by quantizing expert layers to MXFP4 and dense layers to block-FP8.

Ant Ling says a tested Hopper setup moved from 8 GPUs to 4 as a result.

In separate tests, Ant Ling reports an average of 604ms to first token on 8 GPUs and about 1.36 queries per second on 4 GPUs. The workloads differed between the two tests, so the figures are not a direct comparison.

Written by AI from the articles below · updated Oct 14, 11:48 AM ET

Check the sources:

Latest developmentOct 14, 11:19 AM ET
Ant Ling cuts local model weights from 534GB to 292GB with MXFP4 and FP8

Article timeline

The articles in this story. Times are ET.

Oct 14
  1. Ant LingOfficial
    Ant Ling cuts local model weights from 534GB to 292GB with MXFP4 and FP8

    AIAnt Ling says MXFP4 experts and block-FP8 dense layers reduce weights from about 534GB to about 292GB for local serving. A tested Hopper setup moved from 8 GPUs to 4. In separate tests, 8 GPUs averaged 604ms to first token, and 4 GPUs reached about 1.36 QPS, though the workloads differed.

Heat trend

Not enough continuous observations to show a trend yet.