Skip to content
Trending storyDeveloping

MaxxBench tests coding models on modified DeepSWE tasks

1 article1 sourcesince Nov 7Last article 1h ago ·

Overview

AISummary of 1 article

loresjpg introduces MaxxBench, a set of 89 DeepSWE tasks with small changes to the instructions or starting code.

Deedy, posting on X, says the benchmark shows most models are severely benchmaxxed. He reports that models clustered around 74% on DeepSWE end up 23 percentage points apart on MaxxBench, and that Opus 5.5 was the only model whose score rose on the modified tasks.

Written by AI from the articles below · updated Nov 7, 12:15 PM ET

Check the sources:

Latest developmentNov 7, 11:44 AM ET
MaxxBench tests coding models on modified DeepSWE tasks

Article timeline

The articles in this story. Times are ET.

Nov 7
  1. DeedyX
    MaxxBench tests coding models on modified DeepSWE tasks

    AIDeedy says Opus 5.5 was the only model whose DeepSWE score rose when small task changes were made, calling most models severely benchmaxxed. loresjpg introduces MaxxBench, 89 DeepSWE tasks with slight changes to instructions or starting code, reporting that models clustered around 74% on DeepSWE end up 23 percentage points apart.

    Image from @deedydas's post

Heat Trend

Not enough continuous observations to show a trend yet.