Love to see error bars on benchmarks
It’s very much a chart crime not to show them
Hamel Husain says benchmark charts should show error bars
AIcalling their omission a chart crime. He is reacting to Terminal-Bench 4.1, where Ryan Marten says re-grading the 4.0 leaderboard trials with the 4.1 verifiers flipped about 1% of rewards and leaderboard scores remain within previously reported confidence intervals.
Hamel HusainVerified on X
@HamelHusain
Terminal-Bench 4.1 is here with updated verifiers for all tasks. In this release, we focused on: - resisting reward hacking - reducing false negatives - improving reproducibility Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping. Leaderboard scores remain within previously reported confidence intervals.
Source details
Hamel Husain · x.comPublished · added here
Original title: Love to see error bars on benchmarks