5 | ByteDance
Making Frontier Open Models Fast on Hopper with SGLang
Ho-Ren (Jack) Chuang shares the attention, MoE, and speculative decoding work behind faster open-model inference on Hopper.
ByteDance explains faster open-model inference on Hopper with SGLang
AIByteDance engineer Ho-Ren (Jack) Chuang shares the attention, MoE, and speculative decoding work behind faster open-model inference on Hopper GPUs using SGLang. The post covers these three areas, which the source does not quantify with speedup figures.
@lmsysorg
Part of a thread · earlier post