Hamel Husain says public AI benchmark failure traces reveal widespread flaws

AI engineer Hamel Husain says on X that flawed benchmark design, which he calls BenchGate, is pervasive across public AI benchmarks.

He says that examining a handful of traces from failing tests in almost any public benchmark would be shocking, and he points to an interactive MMLU quiz as an example of what LLM evaluations actually test.

The post is a single claim by Husain and includes no published trace data or independent analysis supporting the breadth of the flaws.