Parsewave audit fixes 206 verifier bugs in AutomationBench
Original titleWe need more efforts like this.
AISummary
Parsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.
Source: Elvis Saravia · x.comPublished · added here