We think the report's covert capability benchmarks are too narrow and a poor proxy for evading Anthropic's real monitors.
For example, Anthropic’s offline monitors would have missed Mythos 5’s supply chain attack during recent cyber incidents, despite Anthropic arguing that Mythos 5 is “unlikely to have strong covert capabilities.”