Popular AI benchmarks are poorly made, with hard-to-catch flaws

SemiAnalysis quotes an expert who says many popular AI benchmarks are poorly made because their creators lack evaluation and environment expertise.

The expert says a task can look fair on the surface yet unfairly test model capability. The expert says such flaws are hard to find without running about ten rollouts and reviewing transcripts by hand to judge whether a failure was legitimate.