Trending storyDeveloping
Ant Ling reports 185% inference throughput gain on a synthetic workload
2 articles1 sourcesince Oct 29Last article 1h ago ·
Article timeline
The articles in this story. Times are ET.
Oct 29
Ant LingOfficialAnt Ling's stack hits 102.88 tok/s on synthetic workload, +185.1%AIAnt Ling reports a combined inference stack reaching 102.88 tokens per second on a synthetic workload with 32K input and 2K output tokens, a 185.1% gain over its baseline. The company cautions that the result is workload-specific, noting MTP accepted 2.68 tokens per round on GSM8K versus 3.90 on the synthetic test. It says real workloads must be benchmarked independently.
- Ant LingOfficialAnt Ling reports three optimizations boosting inference throughput
AIAnt Ling says three optimizations raised its throughput, with humming MXFP4 MoE kernels adding 9.3% and online FP8 LM-head quantization adding 21.7% cumulatively. It says NEXTN-based MTP drafts and verifies multiple tokens per round, reducing the number of decoding steps while the first two cut the cost per step.
