Skip to content
Trending storyDeveloping

CRBugBench: Carnegie Mellon and Microsoft benchmark tests AI code review agents on known bugs

1 article1 sourcesince Oct 13Last article 1h ago ·

Overview

AISummary of 1 article

Carnegie Mellon and Microsoft researchers built CRBugBench, a benchmark of 1,800 code review tasks where each pull request introduced a bug that humans confirmed and fixed.

The best agent tested, claude-opus-5 with copilot-cli, caught 71.9% of bugs within its top five comments, according to a paper reported by Rohan Paul on X. The same report says that rate rose from 47.7% for opus-4.6 five months earlier.

The paper concludes that this is still too few bugs for AI to remove human reviewers, and performance dropped on one company's private repositories, where bugs more often need project context.

Written by AI from the articles below · updated Oct 13, 12:22 PM ET

Check the sources:

Latest developmentOct 13, 12:08 PM ET
CRBugBench finds the best code review agent catches 71.9% of known bugs in its top five comments

Article timeline

The articles in this story. Times are ET.

Oct 13
  1. Rohan PaulX
    CRBugBench finds the best code review agent catches 71.9% of known bugs in its top five comments

    AICarnegie Mellon and Microsoft researchers built CRBugBench, a benchmark of 1,800 code review tasks where each pull request introduced a bug that humans confirmed and fixed. The best agent, claude-opus-5 with copilot-cli, caught 71.9% of bugs in its top five comments, up from 47.7% for opus-4.6 five months earlier. Performance dropped on one company's private repos, where bugs more often need project context.

    Image from @rohanpaul_ai's post

Heat trend

Not enough continuous observations to show a trend yet.