Skip to content
Trending storyDeveloping

Hamel Husain says AI benchmark charts should show error bars

1 article1 sourcesince Nov 11Last article 3h ago ·

Overview

AISummary of 1 article

Hamel Husain says benchmark charts should show error bars and calls leaving them out a chart crime.

He made the point in response to Terminal-Bench 4.1. Ryan Marten says re-grading the Terminal-Bench 4.0 leaderboard trials with the 4.1 verifiers flipped about 1% of rewards, and the leaderboard scores remain within previously reported confidence intervals.

Written by AI from the articles below · updated Nov 11, 3:08 AM ET

Check the sources:

Latest developmentNov 11, 2:47 AM ET
Hamel Husain says benchmark charts should show error bars

Article timeline

The articles in this story. Times are ET.

Nov 11
  1. Hamel HusainX
    Hamel Husain says benchmark charts should show error bars

    AIcalling their omission a chart crime. He is reacting to Terminal-Bench 4.1, where Ryan Marten says re-grading the 4.0 leaderboard trials with the 4.1 verifiers flipped about 1% of rewards and leaderboard scores remain within previously reported confidence intervals.

Heat Trend

Current comparable heat
9
Peak comparable heat
10Nov 11, 4:00 AM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.