Tsinghua paper introduces TokenRouter, a serving system for token-level LLM routing

Tsinghua researchers introduce TokenRouter, a serving system for routing individual tokens between small and large language models.

The authors report throughput 2.01 to 64.15 times higher than the stronger existing setup across five routing methods, according to a post by Rohan Paul (@rohanpaul_ai) on X.

The post says current frameworks such as vLLM and SGLang run one model per request, so models sharing an answer wait on each other at every step. TokenRouter instead gives each model its own server, passes partial answers between them, keeps the KV cache, and briefly holds requests to batch work.