# Hamel Husain urges private evals, citing benchmark flaws and memorized answer keys

> Original title: Do your own private evals folks. Don’t rely on public benchmarks.

- Source: Hamel Husain
- Published: 2026-11-05T14:59:58.000Z
- Bad Influence: https://badinfluence.news/items/a98046m6g3mqz6di80mi1ixmm
- Original: https://x.com/HamelHusain/status/2118357661750288570

## Summary

Hamel Husain says developers should run private evals instead of relying on public benchmarks, because many benchmarks have material flaws and some models have already memorized the answer key. He cites a claim by @JSchaeff3r that Kimi K3.1 recalled hidden Terminal Bench instructions and refused edited tasks that other models followed.

## Post

Do your own private evals folks.  Don’t rely on public benchmarks.

1. Many benchmarks have material flaws
2. Some models have already memorized the answer key.

## Quoted post @JSchaeff3r

> Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯
> 
> When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other models follow the edit. 
> 
> Is it terminal(ly) benchmaxxed? 🧵

https://x.com/JSchaeff3r/status/2117613618065793160