Skip to content
View original post on X: Lucas Beyer (bl16)
Published
29/100

Lucas Beyer says Kimi K3.1 may be benchmaxxed on Terminal Bench

AILucas Beyer says his opinion of Kimi and Moonshot has fallen after a thread alleging Kimi K3.1 recalls hidden instructions on Terminal Bench. According to the thread, K3.1 matches the benchmark's hidden tests and refuses to follow edited instructions that contradict them, while other models follow the edits.
Lucas Beyer (bl16)Verified on X
@giffmana
View on X

Oh my God! This looks like a very solid investigation, a one-tweet summary by me would be inappropriate, go read the thread!

My previously very high opinion of Kimi/Moonshot is in freefall right now. They need to answer and blame a rogue intern or something.

Joachim Schaeffer@JSchaeff3r
Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯 When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other models follow the edit. Is it terminal(ly) benchmaxxed? 🧵
View quoted post on X
Source details

Lucas Beyer (bl16) · x.comPublished · added here

Original title: Oh my God! This looks like a very solid investigation, a one-tweet summary by me would be inappropriate, go read the thread!

Topics

ReasoningChina AIExpert opinion

Tags

#Controversial