Nvidia proposes ranking base models by the first passing edit in agent runs

Nvidia proposes scoring base models on the first edit that makes tests pass in replayed successful agent runs.

Researchers replay a strong agent's run, rerun the tests after each edit, and find the first edit that passes. They then test whether each base model favors that edit over failing ones and can write a passing one.

According to Rohan Paul's X post summarizing the paper, how likely each base model was to write that edit ranked 10 models in close agreement with their post-trained SWE-bench Verified scores, at 0.964. The method needs only a benchmark's successful agent runs and its tests.