Skip to content
Trending storyDeveloping

SGLang launches SGLang-TorchTPU, a TPU plugin for serving open models

2 articles2 sourcessince Oct 21Last article 1h ago ·

Overview

AISummary of 2 articles

SGLang launches SGLang-TorchTPU, a TPU plugin for SGLang, built with Google's TPU team and Meta's PyTorch team, according to an SGLang post on X. The plugin is available on GitHub at github.com/sgl-project/sglang-torchtpu.

The post lists Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash and MiniMax H3 as supported open models. It includes a custom MegaKernel, and SGLang says Muse Spark Open reaches 770 tok/s decode at batch size 1 on InferenceX 8K/1K.

LMSYS Org repeated the announcement on X and added that its write-up covers the plugin's features, model coverage and early training exploration on TPU.

Written by AI from the articles below · updated Oct 21, 1:42 PM ET

Check the sources:

Latest developmentOct 21, 1:17 PM ET
LMSYS launches SGLang-TorchTPU to serve open models on TPUs

Article timeline

The articles in this story. Times are ET.

Oct 21
  1. LMSYS OrgOfficialPick
    LMSYS launches SGLang-TorchTPU to serve open models on TPUs

    AILMSYS Org announces SGLang-TorchTPU, a TPU plugin for SGLang built with Google's TPU team and Meta's PyTorch team. The plugin supports models including Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, and MiniMax H3, and the post reports 770 tok/s decode at bs=1 for Muse Spark Open on InferenceX 8K/1K. The project is available on GitHub, and a write-up covers its features, model coverage, and early training exploration on TPU.

  2. SGLangOfficialPick
    SGLang launches SGLang-TorchTPU, a TPU plugin for serving open models

    AISGLang has launched SGLang-TorchTPU, a TPU plugin for SGLang built with Google's TPU team and Meta's PyTorch team. The plugin lists Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, and MiniMax H3 among supported open models. It also includes a custom MegaKernel, with Muse Spark Open reaching 770 tok/s decode at bs=1 on InferenceX 8K/1K, and is available on GitHub.

    Image from @sgl_project's post

Heat Trend

Not enough continuous observations to show a trend yet.