On a synthetic workload with 32K input and 2K output tokens, the combined stack reached 102.88 tok/s—+185.1% over the baseline.
But this is workload-specific. On GSM8K, MTP accepted 2.68 tokens per round, versus 3.90 on the synthetic test. Real workloads must be benchmarked independently.
Ant Ling's stack hits 102.88 tok/s on synthetic workload, +185.1%
AIAnt Ling reports a combined inference stack reaching 102.88 tokens per second on a synthetic workload with 32K input and 2K output tokens, a 185.1% gain over its baseline. The company cautions that the result is workload-specific, noting MTP accepted 2.68 tokens per round on GSM8K versus 3.90 on the synthetic test.
@AntLingAGI
Part of a thread · earlier post
Source details
Ant Ling · x.comPublished · added here
Original title: On a synthetic workload with 32K input and 2K output tokens, the combined stack reached 102.88 tok/s—+185.1% over the baseline.