Without the full logs/traces, I'm not sure what to make of this.
The bigger takeaway for me is the urgency of establishing a standard, more transparent, and reproducible way to report security-related issues involving AI models.
Nothing against Cohere, but there has to be a better way to report these things.
Regarding unexpected tool calling and reasoning, you can’t reliably trust these models to produce what you expect, especially not in those types of tests. That goes for all open-weight and frontier models today. I have spent a lot of time looking at traces to understand this.
Are people not looking at agent traces at all, or what? Lots of weird things happen in agentic coding.
I have also seen weird stuff when you prompt models to do security-related or research tasks related to sandboxes. Both the HF incident and this report are not surprising to me as a dev in this space. That's why I keep raising awareness of the importance of guardrails, evals, and continuous monitoring.
Worth a read still.
