Three optimizations made the difference:
1. Humming MXFP4 MoE kernels: +9.3%
2. Online FP8 LM-head quantization: +21.7% cumulatively
3. NEXTN-based MTP: drafting and verifying multiple tokens per round
The first two reduce the cost per step. MTP reduces the number of steps.
Ant Ling reports three optimizations boosting inference throughput
AIAnt Ling says three optimizations raised its throughput, with humming MXFP4 MoE kernels adding 9.3% and online FP8 LM-head quantization adding 21.7% cumulatively. It says NEXTN-based MTP drafts and verifies multiple tokens per round, reducing the number of decoding steps while the first two cut the cost per step.
@AntLingAGI
Part of a thread · earlier post
Source details
Ant Ling · x.comPublished · added here
Original title: Three optimizations made the difference: