Impressive paper from CMU.
Soon, we will have coding agents that can tune the entire stack, from prompts to token serving.
This work experiments with coding agents to tune an LLM serving stack.
They get Claude Sonnet 5.5 to build a serving system for Gemma E4B that delivers 4.87x the value per GPU dollar of the vLLM reference.
ServerBench asks coding agents to design a complete, specialized serving system and decide how many concurrent sessions to run. The score is latency-qualified token value per dollar of GPU cost, measured by replaying real coding sessions with their dependencies intact. An independent check verifies the system works.
Across 11 coding-agent models and three deployments, 22 of 33 runs produced a verified system and 13 beat the strongest baseline.
The best designs raised capacity and chose the load to match it. Failed runs often shipped optimizations whose new code paths were never tested.
https://arxiv.org/abs/2611.06841
Chat with Paper: https://academy.dair.ai/papers/serverbench-how-can-your-coding-agents-design-specialized-llm-serving-systems-2611.06841