Co-Design Bench: Alibaba and university researchers test AI coding agents on real change requests
Overview
Researchers from Alibaba and several Asian universities built Co-Design Bench, a benchmark of real user change requests for coding agents.
In tests of ten models, the best, GPT-6 Astra, meets about 70 percent of task requirements on average but is fully correct in only 26.4 percent of tasks.
The authors say performance drops as tasks combine more types of changes, and that results also depend on the harness the model runs in.
Written by AI from the articles below · updated Nov 8, 5:14 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- The DecoderNewsCo-Design Bench shows AI coding agents fully finish only about one in four real tasks
AIResearchers from Alibaba and several Asian universities built Co-Design Bench, a benchmark of real user change requests for coding agents. GPT-6 Astra, the best of ten models tested, meets about 70 percent of task requirements on average but is fully correct in only 26.4 percent of tasks. The authors say performance drops as tasks combine more types of changes, and that results also depend on the harness the model runs in.
Heat Trend
Not enough continuous observations to show a trend yet.