Do your own private evals folks. Don’t rely on public benchmarks.
- Many benchmarks have material flaws
- Some models have already memorized the answer key.
Hamel Husain urges private evals, citing benchmark flaws and memorized answer keys
AIHamel Husain says developers should run private evals instead of relying on public benchmarks, because many benchmarks have material flaws and some models have already memorized the answer key. He cites a claim by @JSchaeff3r that Kimi K3.1 recalled hidden Terminal Bench instructions and refused edited tasks that other models followed.
Hamel HusainVerified on X
@HamelHusain
Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯 When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other models follow the edit. Is it terminal(ly) benchmaxxed? 🧵
Source details
Hamel Husain · x.comPublished · added here
Original title: Do your own private evals folks. Don’t rely on public benchmarks.