BenchGate still exists and is very pervasive.
Look at a handful of traces from failing tests in almost any public benchmark you will be shocked.
Hamel Husain says public AI benchmark failure traces reveal widespread flaws
AIHamel Husain says BenchGate, flawed benchmark design, remains pervasive across public benchmarks. He says examining a handful of traces from failing tests in almost any public benchmark would be shocking.
Hamel HusainVerified on X
@HamelHusain
https://d.erenrich.net/are-you-smarter-than-an-llm/index.html Great way to try out MMLU and get a sense for just what, exactly, we are using to evaluate LLMs! I doubt folks would knife fight for a percent on this benchmark if its contents were realized more broadly.
Source details
Hamel Husain · x.comPublished
Original title: BenchGate still exists and is very pervasive.