# KohakuLab open-sources Tokenomos, a faster frontend for vLLM small-model inference

- Source: X search: AI open-source releases
- Published: 2026-10-31T09:18:59.000Z
- Bad Influence: https://badinfluence.news/items/e8uqq85q0leqfmwd6ygvjfuzr
- Original: https://x.com/KBlueleaf/status/2116459910347313274

## Summary

KohakuLab's KBlueleaf says Tokenomos is now open source and reduces frontend overhead, including scheduling and KV cache management, in vLLM. The author reports nearly 3x speedup over vLLM on a small model, measured on an RTX 5090, with the model code borrowed from vLLM. The post's diagram shows a host-loop step time of 0.12 ms at 8 rows and 0.41 ms at 512 rows, with GPU idle time of 0.23 ms per decode step at 512 rows, against 14.37 ms for vLLM.

## Why it matters

The author explains that the speedup over vLLM comes from a rewritten frontend rather than model code, which clarifies where small-model inference overhead lies.

## Post

https://github.com/KohakuBlueleaf/Tokenomos

Tokenomos is open sourced
It resolved the frontend (scheduler and kv cache/state manager system for continuous batching) overhead in vLLM and make small model inference few time faster even with batch size 512 on RTX5090

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHV8sqDRacAAHzv-.jpg&mode=thumb&exp=1793664000&sig=67d610f020acf292)

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHV8sqxQbwAAoX9I.jpg&mode=thumb&exp=1793664000&sig=d1a8b7eea68c2667)

## Quoted post @KBlueleaf

> These number are from RTX5090
> Guess how I get nearly 3x speed over vLLM on small model?
> 
> Better GPU kernel? Weirdo quantize trick?
> NO, all the model code are fully borrow from vLLM. All I did is just FULLY rewrite the frontend and now the CPU is no longer the bottleneck

https://x.com/KBlueleaf/status/2116202706671153245