Shopify Turns Open Small Models into "Task Experts": GraphQL Sub-Agent Slightly Outperforms Frontier Models, 7x Faster and ~90% Cheaper
From a summary by @Drewch of the @ShopifyEng team on the NVIDIA GTC Berlin 2026 talk. Shopify's e-commerce assistant Sidekick uses a division of labor where "frontier models serve as the brain, open models serve as the experts": frontier models handle conversations with merchants, planning, and tool calls; the heavy work that is read-heavy and write-light goes to open models that Shopify post-trains and deploys itself. On GraphQL tasks, they lead on quality, speed, and cost.
https://shopify.engineering/open-model-task-experts
Why split the work this way
The GraphQL sub-agent has to read the API schema, look up docs, write queries, execute them, and return results, reading about 40 tokens for every token it outputs. Using a frontier model's API means paying for every token, so context-heavy tasks like this are best handed to small models that you host yourself.
The approach: a repeated model-optimization flywheel
1. Define quality first: product experts write a scoring rubric (completeness, executability, answer quality). Using the agreement between two experts as the benchmark, they optimize an LLM judge with GEPA and ACE until it agrees with human labels as well as the experts agree with each other. From then on, every change is scored by this judge.
2. First, tune the frontier-model agent to its best: run an autoresearch loop that makes one small change at a time to the prompt, tool descriptions, or control flow, and keeps it only if the score goes up. The tuned frontier agent becomes the baseline the open model has to beat.
3. Choose a model: no pretraining. Drop candidate models into the real sub-agent and test them with the same tools and judge. They ended up picking Nemotron 3.5 Lightning, a MoE model with 30B total parameters and 3B active per token.
4. Test inference performance from the start: a model that can't run in production isn't worth training. They replay production traffic with AIPerf (using synthetic text in place of customer data) and use the results to choose the inference framework.
5. Turn failure cases into training data (self-healing): each night, the judge samples production conversations. Imperfect ones are sent to a frontier model acting as a "critic" that gives hints but not answers; then a simulated merchant reruns the conversation with the hint. Once it passes, the hint is removed, the conversation is anonymized, and it's added to the training set. Over the most recent 86 nights, 59,535 conversations were repaired, 69% of the failure cases attempted, with the rest fixed by humans.
6. Post-training: train with Megatron Bridge on Blackwell GPUs, then rescore and move to the next round.
Results (GraphQL task)
· Judge score: out of the box Nemotron scored 37.7, versus 73 for the frontier model. After the first training round it reached 61.5, and after 6 rounds and 61,000 repaired samples it reached 74.5.
· Speed: median response 0.87 seconds versus 6.50 seconds for the frontier model API; the slowest 1% of requests are 8x faster.
· Cost: about 90% lower than the frontier model. When the post-trained models are self-hosted, cost is 90% to 98% lower than frontier model list pricing.