KohakuLab open-sources Tokenomos, a rewritten vLLM frontend for small-model inference

KBlueleaf of KohakuLab says Tokenomos, an open-source rewrite of vLLM's frontend scheduler and KV cache/state manager, cuts the CPU-side overhead that slows continuous batching.

In a post on X, the author reports nearly 3x speedup over vLLM on a small model at batch size 512 on an RTX 5090. The model code is borrowed from vLLM, and the author says the change is a full rewrite of the frontend only.

The post's diagram gives a host-loop step time of 0.12 ms at 8 rows and 0.41 ms at 512 rows, and 0.23 ms of GPU idle time per decode step at 512 rows, against 14.37 ms for vLLM. The author states that the CPU is no longer the bottleneck. These figures are the author's own measurements and have not been independently verified.