5 | ByteDance
Making Frontier Open Models Fast on Hopper with SGLang
Ho-Ren (Jack) Chuang shares the attention, MoE, and speculative decoding work behind faster open-model inference on Hopper.
ByteDance explains faster open-model inference on Hopper with SGLang
AIByteDance engineer Ho-Ren (Jack) Chuang describes the attention, MoE, and speculative decoding work behind faster open-model inference on Hopper GPUs using SGLang. The post shares the techniques without giving benchmark figures or specific speedups.
@lmsysorg
Part of a thread · earlier post