370 tokens/sec with MiniCPM-V 4.7?! 🔥
Love seeing the community push the limits of what's possible with speculative decoding.
Huge shoutout to @Tech2Wild for putting this together! 🚀
MiniCPM-V 4.7 hits 370 tokens/sec with DSPARK speculative decoding
AISummary
Tech2Wild reports that OpenBMB's MiniCPM v4.7 35B A3B, paired with a DSPARK drafter, sustained 370.0 tokens/sec aggregate in a 10-second test. The run produced 3,589 tokens with a 0.30-second average time to first token and a 10-second average end-to-end latency. OpenBMB's post celebrates the result as a community speculative decoding achievement.
Post on XView on X
OpenBMBVerified on X
@OpenBMB
So @OpenBMB MiniCPM v4.7 35B A3B Comes With a DSPARK Drafter. This Thing SPITS ! 370 tok/s • Total: 3,589 tokens in 10s • Aggregate: 370.0 tok/s sustained · 359.0 wall-avg · 351.8 peak burst • Per-stream decode: 370.0 high / 370.0 low / 370.0 avg tok/s • TTFT 0.30s avg · E2E 10s avg
Source: OpenBMB · x.comPublished · added here