# Joel Niklaus releases the Harness Leaderboard comparing nine coding-agent harnesses

> Original title: Such an awesome, empirically rigorous benchmark!! 🙌

- Source: Paige Bailey
- Published: 2026-11-07T00:46:44.000Z
- Bad Influence: https://badinfluence.news/items/tl0nmesmdxfssbpg5fn846fv9
- Original: https://x.com/DynamicWebPaige/status/2118867715159126269

## Summary

Joel Niklaus released the Harness Leaderboard, which compares nine coding-agent harnesses on open models across 28,849 trials. Averaged over three models, mini-swe-agent solves 51.4% of tasks and Pi reaches 49.4% at $0.29 per solved task, about 45% less than mini-swe-agent. Paige Bailey, quoting the post, praised the benchmark and said the harness matters as much as the model.

## Post

Such an awesome, empirically rigorous benchmark!! 🙌

Holding the model fixed and still seeing a ~2x swing in solve rate -- and ~11.7x difference in cost per solved task -- is proof of how much the harness actually matters. (And Harness Lens documenting prompts, tools, and compaction across 34 harnesses is 😍.)

Would love to see this exact methodology run on frontier proprietary models across open harnesses next! 👀

## Quoted post @joelniklaus

> I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
> 
> Which harness should you run your open model in? I ran 28,849 trials to find out.
> 
> The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
> 
> The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
> 
> Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
> 
> One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
> 
> The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
> 
> @opencode costs $3.35 per solved task, 11.7 times as much as Pi.
> 
> See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
> 
> Leaderboard: https://huggingface.co/spaces/joelniklaus/harness-leaderboard
> Harness Lens: https://huggingface.co/spaces/harness-hub/harness-lens

https://x.com/joelniklaus/status/2118373808692437403