# OpenRouter explains how to benchmark LLMs with code-graded, app-specific tasks

> Original title: How to Benchmark LLMs

- Source: OpenRouter Blog
- Published: 2026-11-12T00:00:00.000Z
- Bad Influence: https://badinfluence.news/items/dthohjibcf9t8gl2huqe4i6rb
- Original: https://openrouter.ai/blog/tutorials/how-to-benchmark-llms/

## Summary

OpenRouter says LLM choices should rest on tasks drawn from an application's own inputs and output formats, graded by code. In its 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials, DeepSeek had the lowest response cost at 157/180 passes, and the public ranking agreed weakly with a Spearman correlation of 0.109. The companion Python example runs offline checks, but the full-study sandbox supports Windows x64 only.