Skip to content
Trending storyDeveloping

Parsimony: open-source benchmark ranks AI models by net code added

1 article1 sourcesince Oct 13Last article 4h ago ·

Overview

AISummary of 1 article

Parsimony, an open-source benchmark released by lasu, measures how much code AI models add to patches, counting net coding units with deleted code subtracted rather than raw lines.

It ranks models by mean net units added on SWE-bench Verified tasks, lowest first, and includes failed attempts in the mean. The creator says existing benchmarks measure solve rate and cost but do not penalize excess code, which makes a codebase harder to work with long term. Only Python is measured, and the benchmark does not rerun tests.

Written by AI from the articles below · updated Oct 13, 7:21 PM ET

Check the sources:

Article timeline

The articles in this story. Times are ET.

Oct 13
  1. Hacker News · Show HN, AI (20+ points)Blog
    Parsimony benchmarks how much code AI models add to patches

    AIParsimony, an open-source benchmark, measures how much code AI models add by counting net coding units, with deleted code subtracted, instead of lines. It ranks models by mean net units added on SWE-bench Verified tasks, lowest first, and counts failed attempts in the mean. Only Python is measured, and the benchmark does not rerun tests.

Heat trend

Not enough continuous observations to show a trend yet.