📱 35B parameters. Real-time inference. On a phone.
Meet Lookahead Expert Scheduling (LES), the inference system powering MiniCPM-V 4.7’s 35B-A3B MoE model on edge devices.
🚀 On an iPhone 18 Pro Max, the full system reaches 20.51 tokens per second, a 6.07× speedup over loading expert weights on demand.
How? Three coordinated optimizations:
- 🔮 Scheduling: Predict and prefetch the next layer’s experts while the current layer computes.
- 💾 Storage: Combine contiguous expert layouts with mixed-precision quantization to reduce loading overhead.
- ⚡ Compute: Use DSpark speculative decoding to verify multiple candidate tokens in one pass.
Predictions guide prefetching; the model’s actual router still determines which experts execute. The architecture, expert counts, and routing remain intact. Mixed-precision quantization reduces the weight file by 26.7%, with just a 0.16-point drop in the five-benchmark average.
LES shows how better inference systems can bring larger, more capable models to everyday devices. 🌍
👉 LES: https://github.com/openbmb/llama.cpp-omni/blob/main/docs/les/lookahead-expert-scheduling.md
🔗 GitHub: http://github.com/OpenBMB/MiniCPM-V