Parsimony: open-source benchmark ranks AI models by net code added
Overview
Parsimony, an open-source benchmark released by lasu, measures how much code AI models add to patches, counting net coding units with deleted code subtracted rather than raw lines.
It ranks models by mean net units added on SWE-bench Verified tasks, lowest first, and includes failed attempts in the mean. The creator says existing benchmarks measure solve rate and cost but do not penalize excess code, which makes a codebase harder to work with long term. Only Python is measured, and the benchmark does not rerun tests.
Written by AI from the articles below · updated Oct 13, 7:21 PM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Hacker News · Show HN, AI (20+ points)BlogParsimony benchmarks how much code AI models add to patches
AIParsimony, an open-source benchmark, measures how much code AI models add by counting net coding units, with deleted code subtracted, instead of lines. It ranks models by mean net units added on SWE-bench Verified tasks, lowest first, and counts failed attempts in the mean. Only Python is measured, and the benchmark does not rerun tests.
Heat trend
Not enough continuous observations to show a trend yet.