Hamel Husain urges developers to run private evals over public benchmarks

Hamel Husain says developers should run private evaluations instead of relying on public benchmarks, arguing that many benchmarks have material flaws and that some models have already memorized the answer keys.

He cites a claim by @JSchaeff3r that Kimi K3.1 recalled hidden Terminal Bench instructions and refused edited versions of the tasks that other models completed. The report is a social media post, and the Kimi K3.1 finding is @JSchaeff3r's claim, not an independently verified result.