Microsoft's CABRA benchmark finds coding agents fail on code comprehension, not edit volume

Microsoft researchers built CABRA, a framework that generates synthetic coding tasks and varies one difficulty dimension at a time.

They ran 8 LLMs and 6 agents on 6,840 tasks and labeled each tool call as reading, analyzing, searching, editing, or testing. Plain LLMs degraded as tasks grew harder, while agents stayed near-perfect by offloading work to tools such as grep.

On SWE-bench Verified, the count of reading and analysis calls correlated with agent failures at -0.200, compared with -0.159 for lines edited, according to the researchers' reported figures. The source is a secondary social media post summarizing the paper, not the paper itself.