Probably great at defensive cyber too, but no one will post that and get clicks!
Lambert says models that excel at offense likely excel at defense too
AINathan Lambert says models that are probably strong at defensive cybersecurity will not get attention because defense posts draw fewer clicks. The post is a comment on the quoted Vals AI benchmark, which found DeepSeek V4.1 Flash 1017 beat GPT 6 Sol on a multi-step offensive cybersecurity task.
Nathan LambertVerified on X
@natolambert
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program. We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
Source details
Nathan Lambert · x.comPublished · added here
Original title: Probably great at defensive cyber too, but no one will post that and get clicks!