# Unsloth reports DeepSeek GGUF quants lead on 512K-context agentic benchmarks

> Original title: We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD3.1 GGUFs and show we're >10-20% better ...

- Source: Daniel Han
- Published: 2026-11-02T15:36:38.000Z
- Bad Influence: https://badinfluence.news/items/c430u1uwndt671byd7scdkx16
- Original: https://x.com/danielhanchen/status/2117279726742233225

## Summary

Unsloth says its DeepSeek-V4.1-Flash-1017 GGUFs score more than 10% higher than GSQ-RCO on DeepSWE, Terminal Bench, Automation Bench, HLE, and SciCode at 512K context. The 1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal Bench, and 52.7 on Automation Bench. Unsloth also re-measured GSQ-RCO's recovery figures, reporting lower results than GSQ-RCO claims for its IQ1_M and IQ2_XXS quants.

## Post

We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD3.1 GGUFs and show we're >10-20% better than GSQ-RCO

We ran all other benchmarks (14 in total) to 128K, fixed grading, infra. AIME2025 64x, DeepSWE/TB 3x, gpt-6-luna xhigh for HLE

We addressed some issues people mentioned with these new benchmarks:

1. 32K context was too short - we did 512K context for agentic + 128K for general tasks

2. Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench

3. Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See https://unsloth.ai/docs/models/deepseek-v4.1-flash#benchmark-runs-per-model where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.

4. Our graders are wrong - we used Math Verify https://huggingface.co/blog/math_verify_leaderboard, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest.

See https://unsloth.ai/docs/models/deepseek-v4.1-flash#graders and https://unsloth.ai/docs/models/deepseek-v4.1-flash#infrastructure-and-benchmark-issues-and-fixes for infra fixes

For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over).

Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues

We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013.

GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22].

GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001.

If we can get more details on how the benchmarks were run, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026

## Quoted post @UnslothAI

> We benchmarked DeepSeek-V4.1-Flash-1017 GGUFs on Terminal Bench, DeepSWE & more on 512K context. Unsloth ranks highest overall.
> 
> 1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.
> 
> GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

https://x.com/UnslothAI/status/2117266365929365525