# Thoroughness versus over-editing makes automated code review subjective

> Original title: Really cool methodology here.

- Source: wh
- Published: 2026-10-27T21:59:05.000Z
- Bad Influence: https://badinfluence.news/items/xu5nub8pxoy1vsw2k7vwt5bn8
- Original: https://x.com/nrehiew_/status/2115201646749954389

## Summary

@nrehiew_ says the line between thorough bug fixing in unprompted code paths and over-editing beyond a prompt's request is very fine. He suggests automated code review design has entered a subjective phase. The post reflects on Cognition's FrontierCode 2.0 benchmark, where the top model scores 45%.

## Post

Really cool methodology here. 

The line between thoroughness (fixing bugs in non-prompt-specified code paths) and over-editing beyond what a prompt asks for is extremely fine. 

Really likely we have started to enter the subjective part of automated code review design now

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHVq0Exva8AAt4JW.jpg&mode=thumb&exp=1793404800&sig=2b4fa923d0845f4d)

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHVq0GrUa8AA_Mkr.jpg&mode=thumb&exp=1793404800&sig=5af45ed138e008fd)

## Quoted post @cognition

> Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
> 
> Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
> 
> FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
> 
> One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
> 
> The result is a benchmark that is far from saturated, with the top model scoring just 45%.

https://x.com/cognition/status/2115132488775524793