Janas-LLM streams MoE model experts from SSD to run models larger than RAM
Overview
Maurizio Cammalleri releases Janas-LLM, an open-source engine written in C that runs mixture-of-experts language models larger than RAM by streaming expert weights from an NVMe SSD. The weights stay on disk, the experts each token needs are read from the SSD, and frequently reused experts stay in memory.
Cammalleri describes the project as an experiment, not a product.
In a Hacker News post, the author reports 37 tokens per second writing from Qwen3-Next-80B, a 48.4 GB model, on a laptop with 32 GB of memory, where llama.cpp reached 2.8 tokens per second. The author notes the benchmarks come from one machine and that other hardware will produce different figures.
Written by AI from the articles below · updated Oct 13, 7:11 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Hacker News · AI (150+ points)BlogPickJanas-LLM streams experts from SSD to run 80B models on a 32 GB laptop
AIJanas-LLM, an open-source engine written in C, runs mixture-of-experts language models larger than RAM by streaming expert weights from an NVMe SSD. The author reports 37 tok/s writing from Qwen3-Next-80B, a 48.4 GB model, on a laptop with 32 GB of memory, where llama.cpp reached 2.8 tok/s. The benchmarks come from one machine, with the author noting that other hardware will produce different figures.
Heat trend
Not enough continuous observations to show a trend yet.