Unsloth publishes benchmark methods amid dispute with GSQ-RCO

AIUnsloth co-founder Daniel Han says the team asked GSQ-RCO for more detail on how its benchmarks were made, and that Unsloth has published its own methods publicly while GSQ-RCO has not. Han says Unsloth ran DeepSWE, Terminal Bench, Automation Bench, HLE, and SciCode at 512K context on its DeepSeek UD3.1 GGUFs, reporting it is more than 10-20% better than GSQ-RCO. He adds that the team used 35.1B tokens of benchmarking, compared with 1.3B before.

Daniel Han

@danielhanchen

View on X

We asked the GSQ-RCO team if we can get more details on how the benchmarks were made

Reddit (now deleted):
"Please just try to reproduce our numbers yourself. Even if your hardware doesn’t allow you to run DeepSeek, you can still check our benchmarks on QFN and Qwen3.8-27B. Our models got popular simply because they’re better than Unsloth’s. And now they’re pissed off about it and doing everything they can to smear us. It’s as simple as that."

"What a natural looking chart. I wonder why you guys even bother creating larger models because it seems the IQ1_M or even IQ1_S model is just the same as base model."

We provided all our benchmarking methods, issues we faced ourselves, and more publically at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-october-30th-2026 - ISTA still hasn't provided their methods.

Daniel Han@danielhanchen

We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD3.1 GGUFs and show we're >10-20% better than GSQ-RCO We ran all other benchmarks (14 in total) to 128K, fixed grading, infra. AIME2025 64x, DeepSWE/TB 3x, gpt-6-luna xhigh for HLE We addressed some issues people mentioned with these new benchmarks: 1. 32K context was too short - we did 512K context for agentic + 128K for general tasks 2. Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench 3. Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See https://unsloth.ai/docs/models/deepseek-v4.1-flash#benchmark-runs-per-model where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more. 4. Our graders are wrong - we used Math Verify https://huggingface.co/blog/math_verify_leaderboard, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest. See https://unsloth.ai/docs/models/deepseek-v4.1-flash#graders and https://unsloth.ai/docs/models/deepseek-v4.1-flash#infrastructure-and-benchmark-issues-and-fixes for infra fixes For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over). Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013. GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22]. GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001. If we can get more details on how the benchmarks were run, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026

View quoted post on X

Source details

Daniel Han · x.comPublished Nov 2, 11:52 AM ET

Original title: We asked the GSQ-RCO team if we can get more details on how the benchmarks were made

Unsloth publishes benchmark methods amid dispute with GSQ-RCO

Topics

Tags