New Carnegie Mellon and Microsoft paper finds that the best AI code reviewer caught 71.9% of known bugs in its top 5 comments, still too few to remove human reviewers.
Agents now write code faster than people can review it, tempting teams to let AI review it too. Older review benchmarks scored AI against human comments, which can be nitpicks or miss real bugs.
They built CRBugBench, 1,800 code review tasks where each pull request introduced a bug that humans confirmed and fixed. The agent reviews the original change in a sandbox, and AI judges check whether its top findings name that bug.
The best agent, claude-opus-5, caught 71.9% of bugs in its top 5 comments, up from 47.7% for opus-4.6 5 months earlier. Performance dropped up to 25.5% on 1 company's private repos, where bugs more often need project context.
Use review agents alongside human reviewers, and give them your repo's own knowledge, especially on large or context-heavy changes.