1 | @LinkedIn
Ads Ranking with SLM and SGLang at LinkedIn
Xianxing Zhang on serving an ads ranker at 10K+ QPS with under 80 ms scoring latency.
LinkedIn describes serving an ads ranker with SLM and SGLang
AIXianxing Zhang explains how LinkedIn serves its ads ranker at more than 10K QPS with under 80 ms scoring latency. The post uses SLM and SGLang for the serving stack.
@lmsysorg
Part of a thread · earlier post