Skip to content
View original post on X: 琥珀青葉@KohakuLab
Published
46/100

KohakuLab open-sources Tokenomos, a faster frontend for vLLM small-model inference

AIKohakuLab's KBlueleaf says Tokenomos is now open source and reduces frontend overhead, including scheduling and KV cache management, in vLLM. The author reports nearly 3x speedup over vLLM on a small model, measured on an RTX 5090, with the model code borrowed from vLLM. The post's diagram shows a host-loop step time of 0.12 ms at 8 rows and 0.41 ms at 512 rows, with GPU idle time of 0.23 ms per decode step at 512 rows, against 14.37 ms for vLLM.

AIWhy it matters.

The author explains that the speedup over vLLM comes from a rewritten frontend rather than model code, which clarifies where small-model inference overhead lies.

琥珀青葉@KohakuLabVerified on X
@KBlueleaf
View on X

https://github.com/KohakuBlueleaf/Tokenomos

Tokenomos is open sourced
It resolved the frontend (scheduler and kv cache/state manager system for continuous batching) overhead in vLLM and make small model inference few time faster even with batch size 512 on RTX5090

琥珀青葉@KohakuLab@KBlueleaf
These number are from RTX5090 Guess how I get nearly 3x speed over vLLM on small model? Better GPU kernel? Weirdo quantize trick? NO, all the model code are fully borrow from vLLM. All I did is just FULLY rewrite the frontend and now the CPU is no longer the bottleneck
View quoted post on X
Source details

琥珀青葉@KohakuLab · x.comPublished · added here

Topics

Open-source ecosystemDeployment & engineeringProduct updates