vLLM releases TorchTPU backend for running vLLM on Google Cloud TPUs
AIvLLM releases vllm-torchtpu, an out-of-tree platform plugin that runs vLLM on Google Cloud TPUs, so most models that already run in vLLM can be served on TPU without model code changes. The plugin keeps vLLM's CLI, OpenAI-compatible server, scheduler, and KV cache manager unchanged, and adds Pallas kernels for attention, Mixture-of-Experts, and quantized matmul.
AIWhy it matters.
The post specifies how the TPU backend reuses vLLM's existing model code, scheduler, and API, which clarifies what changes and what stays the same for users.
Source details
vLLM Blog · Inferact and Google Teams · vllm.aiPublished · added here
Original title: Introducing vLLM TorchTPU