ModelBest explains Lookahead Expert Scheduling for running MoE models on phones
Overview
ModelBest has published an engineering write-up on Lookahead Expert Scheduling (LES), the inference system behind its MiniCPM-V 4.7 35B-A3B model.
The company says LES runs the model on an iPhone 18 Pro Max at over 20 tokens per second.
LES keeps expert weights in flash storage and predicts which experts to load using similarity between adjacent-layer routers, with about 80% candidate recall. According to ModelBest, mixed-precision quantization and DSpark speculative decoding add further speed.
Written by AI from the articles below · updated Nov 9, 9:39 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- PandailyNewsModelBest explains how LES runs a 35B MoE model on phones at over 20 tokens/s
AIModelBest published an engineering write-up on Lookahead Expert Scheduling (LES), the inference system behind MiniCPM-V 4.7 35B-A3B, which it says runs the model on an iPhone 18 Pro Max at over 20 tokens per second. The method keeps expert weights in flash storage and predicts which experts to load using adjacent-layer router similarity, with about 80% candidate recall, while mixed-precision quantization and DSpark speculative decoding add further speed.
Heat Trend
Not enough continuous observations to show a trend yet.