The only model that actually scored higher on DeepSWE when small changes were made to the task, was Opus 5.5
Most models are severely benchmaxxed.
MaxxBench tests coding models on modified DeepSWE tasks
AIDeedy says Opus 5.5 was the only model whose DeepSWE score rose when small task changes were made, calling most models severely benchmaxxed. loresjpg introduces MaxxBench, 89 DeepSWE tasks with slight changes to instructions or starting code, reporting that models clustered around 74% on DeepSWE end up 23 percentage points apart.
DeedyVerified on X
@deedydas
Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good at coding, or good at DeepSWE? 🧵
Source details
Deedy · x.comPublished · added here
Original title: The only model that actually scored higher on DeepSWE when small changes were made to the task, was Opus 5.5