ModelBest explains how LES runs a 35B MoE model on phones at over 20 tokens/s
AIModelBest published an engineering write-up on Lookahead Expert Scheduling (LES), the inference system behind MiniCPM-V 4.7 35B-A3B, which it says runs the model on an iPhone 18 Pro Max at over 20 tokens per second. The method keeps expert weights in flash storage and predicts which experts to load using adjacent-layer router similarity, with about 80% candidate recall, while mixed-precision quantization and DSpark speculative decoding add further speed.
Source details
Pandaily · contact@pandaily.com (Pandaily) · pandaily.comPublished · added here
Original title: ModelBest Details LES, the Method Behind a 35B MoE Model Running on Phones at Over 20 Tokens/s