# Joël Niklaus releases a Harness Leaderboard comparing nine coding-agent harnesses on open models

> Original title: same open model, different harness => very different results

- Source: Thomas Wolf
- Published: 2026-11-07T00:28:13.000Z
- Bad Influence: https://badinfluence.news/items/gcbwpcoo4a6q9gy1a75897yzv
- Original: https://x.com/Thom_Wolf/status/2118863053001531676

## Summary

Joël Niklaus releases the Harness Leaderboard, which compares nine coding-agent harnesses with the model, provider and reasoning effort held fixed. Across 28,849 trials on Terminal-Bench 2.1 and DeepSWE, averaged over three open models, mini-swe-agent solves 51.4% of tasks and OpenClaw 27.3%. Pi solves 49.4% at $0.29 per solved task, about 45% less than mini-swe-agent's $0.53, while opencode costs $3.35 per solved task.

## Why it matters

The leaderboard holds the model fixed and varies only the harness, showing how much the agent wrapper alone changes solve rates and cost per task.

## Post

same open model, different harness => very different results

Joël set out to find which harness actually gets the most out of open models in November 2026

## Quoted post @joelniklaus

> I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
> 
> Which harness should you run your open model in? I ran 28,849 trials to find out.
> 
> The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
> 
> The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
> 
> Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
> 
> One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
> 
> The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
> 
> @opencode costs $3.35 per solved task, 11.7 times as much as Pi.
> 
> See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
> 
> Leaderboard: https://huggingface.co/spaces/joelniklaus/harness-leaderboard
> Harness Lens: https://huggingface.co/spaces/harness-hub/harness-lens

https://x.com/joelniklaus/status/2118373808692437403