Such an awesome, empirically rigorous benchmark!! 🙌
Holding the model fixed and still seeing a ~2x swing in solve rate -- and ~11.7x difference in cost per solved task -- is proof of how much the harness actually matters. (And Harness Lens documenting prompts, tools, and compaction across 34 harnesses is 😍.)
Would love to see this exact methodology run on frontier proprietary models across open harnesses next! 👀
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: https://huggingface.co/spaces/joelniklaus/harness-leaderboard Harness Lens: https://huggingface.co/spaces/harness-hub/harness-lens
This story is featured in top picks: “Joël Niklaus releases a Harness Leaderboard comparing nine coding-agent harnesses on open models”
👩💻 Paige Bailey · x.comPublished · added here
Original title: Such an awesome, empirically rigorous benchmark!! 🙌
