Many of the most popular AI benchmarks are badly made. The flaws are easy to build in and hard to catch.
“A lot of the really popular benchmarks out there are kind of trash. This is largely because they’re made by people who don’t have expertise in making evals or environments or training data in general.”
“It’s very easy to make a task that seems good on a surface level, but it’s actually unfair in some way, such that it’s a poor test of model capabilities.”
“A lot of people make their tasks harder by making them unfair rather than actually making the task difficult.”
“It’s very hard to figure this out unless you run ten rollouts, look at the transcripts, see why the model failed, and determine manually: was this actually a legitimate failure?”