⚡ Lookahead Expert Scheduling: Making On-Device MoE Inference Faster
Running a 35B MoE model on a device with limited memory presents a key challenge: expert weights can be stored on SSD, but loading them on demand can stall generation.
MiniCPM-V 4.7 addresses this with LES, which coordinates expert loading and computation rather than waiting for every required expert to load before continuing.
🔹 Expert prefetch and compute scheduling: LES predicts which experts the next layer may need through cross-layer routing prediction. It prioritizes computation for experts already loaded and runs their computation while loading the remaining required experts in parallel. Predictions guide preloading and scheduling only; the original token-level router still determines the experts used for actual computation. No additional training is required.
🔹 Optimized weight access: Fine-grained, layer-wise mixed-precision quantization reduces weight storage and transfer overhead while preserving model capability. Reorganizing expert weights into contiguous layouts also reduces scattered reads and improves sequential flash-storage access.
Lookahead Expert Scheduling: https://github.com/tc-mb/llama.cpp-omni/blob/master/docs/les/lookahead-expert-scheduling.md