Collinear AI releases open 100-task CWE-bench v2 for AI security-fix evaluation

Collinear AI releases CWE-bench v2, a public set of 100 tasks across 91 CWEs in which agents must find and fix security weaknesses without breaking existing functionality.

The set, grader and harness are open, and the tasks can be run from the Hugging Face Hub with Harbor, according to the company's Hugging Face Blog post. Scoring is fully programmatic, so results are deterministic and reproducible, the company says. The dataset is gated, so users must accept its access conditions on the dataset page first.

Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, ahead of Claude Opus 5.5 at 67.3% and 78.0%, the post reports.