# Many popular AI benchmarks are poorly made, with flaws that are hard to catch

> Original title: Many of the most popular AI benchmarks are badly made. The flaws are easy to build in and hard to catch.

- Source: SemiAnalysis
- Published: 2026-11-05T23:01:20.000Z
- Bad Influence: https://badinfluence.news/items/t8rmh2g4826bc5bw9va2i609w
- Original: https://x.com/SemiAnalysis_/status/2118478804020564228

## Summary

SemiAnalysis quotes an expert who says many popular AI benchmarks are poorly made because their creators lack eval and environment expertise. The speaker says a task can seem fair on the surface yet unfairly test model capability, and that such flaws are hard to catch without running ten rollouts and manually reviewing transcripts to judge whether a failure was legitimate.

## Post

Many of the most popular AI benchmarks are badly made. The flaws are easy to build in and hard to catch.

“A lot of the really popular benchmarks out there are kind of trash. This is largely because they’re made by people who don’t have expertise in making evals or environments or training data in general.”

“It’s very easy to make a task that seems good on a surface level, but it’s actually unfair in some way, such that it’s a poor test of model capabilities.”

“A lot of people make their tasks harder by making them unfair rather than actually making the task difficult.”

“It’s very hard to figure this out unless you run ten rollouts, look at the transcripts, see why the model failed, and determine manually: was this actually a legitimate failure?”

[Video](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Famplify_video_thumb%2F2118477903444795392%2Fimg%2FxdtGqrHyEGdmulSZ.jpg&mode=thumb&exp=1794182400&sig=f22d3c08bb6612c8)