We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD3.1 GGUFs and show we're >10-20% better than GSQ-RCO
We ran all other benchmarks (14 in total) to 128K, fixed grading, infra. AIME2025 64x, DeepSWE/TB 3x, gpt-6-luna xhigh for HLE
We addressed some issues people mentioned with these new benchmarks:
1. 32K context was too short - we did 512K context for agentic + 128K for general tasks
2. Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench
3. Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See https://unsloth.ai/docs/models/deepseek-v4.1-flash#benchmark-runs-per-model where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.
4. Our graders are wrong - we used Math Verify https://huggingface.co/blog/math_verify_leaderboard, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest.
See https://unsloth.ai/docs/models/deepseek-v4.1-flash#graders and https://unsloth.ai/docs/models/deepseek-v4.1-flash#infrastructure-and-benchmark-issues-and-fixes for infra fixes
For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over).
Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues
We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013.
GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22].
GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001.
If we can get more details on how the benchmarks were run, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026