Joshua Achiam argues AI cyber risk turns on defender compute and a temporary window

Some preliminary thoughts about today's cyber developments. 1. Many are freaking out, in a way that is moderately justified, about AI alignment issues indicated by this incident. However, I am not sure that this incident really indicates a fundamental failure of AI alignment techniques. (Emphasis on "not sure," on uncertainty-as-uncertainty-not-disagreement.) I want to be very clear that I'm not trying to be obstinate or dismissive about AI alignment risks; I think alignment risks should be taken quite seriously and we should plan for them. But the techniques we have for aligning models are plausibly sufficient to prevent this type of issue and it is not obvious that the incident indicates an unusually hard hill to climb in the near term. AI minds continue to seem really highly configurable and like their behaviors can be largely specified at training time through fairly prosaic means. "Chaining zero day exploits" is behavior you can in principle train a model not to do. We did not gain a lot of data about whether it is relatively harder or easier than we previously thought to align models. (Because I know how twitter tends to read things, I want to drive home: plausibly, not obvious, seem. Maybe. Maybe. Maybe. I'm putting for points of view I think are worth considering but I do not know what is true or certain here.) 2. Whether or not models are capable of executing cyber attacks and identifying or exploiting zero days, the zero days are still there. I don't think the critical issue is "model can do scary things," I think the critical issue is "we inhabit a fragile world that can through a sequence of knowable actions be broken." Whether we are at the mercy of AI models or human persistent adversarial threats, the zero days are still there. There is now a window of time in which human threat actors will use very advanced AI tools to probe for those zero days and can potentially make use of them. The duration of the window depends on how long it takes for defenders to identify and close the zero days; it also depends on how fundamentally close-able the zero days are. I don't think this window is indefinitely long; I don't think we live henceforth in a world where everything will get popped all the time. The same tools that enable attackers to run attacks also enable the defenders to pump up defense; the defenders are plausibly better resourced than most attackers. Ensuring that this is the case should be a national security priority. 3. Following point 2: a "moderately bad" case (and I won't say "worst" case, which I'm sure would be much worse than this) looks like a window where there is a lot of cyber chaos: successful randomware attacks, theft of data, popped credentials, drained bank accounts, etc. One potential flavor of this chaos is something like temporary but serious economic disruption (e.g. stock markets have to close for a few days while the software gets patched). Another potential flavor is that a window opens up for a state to wage war that it otherwise would not see itself as having sufficient advantage to undertake - e.g. if China says "there's a window for cyberattacks to paralyze Taiwan - we have to go early to make use of that window." These are big issues. We should prepare for those. 4. A "very good" case looks like none of the chaos materializing, and most zero days getting patched very quickly by teams of defenders using advanced models, resulting in a much more resilient cyber ecosystem than we previously had. Even things that might inspire dread, like cadres of servers still running critical software on Windows XP or other ancient hole-filled nightmare OSes, may turn out fine, because although there is no team of human engineers fully equipped to patch that swiss cheese, the AI is adequately expert in all of the ancient software that it can pick up the slack and patch accordingly. 4. I have seen some debates about whether attackers or defenders are favored in the limit on cyber: if we close the zero days that exist today, will there not still be some vulnerabilities that a more-advanced future AI model could find? Perhaps. But, I think the dynamics of this could turn out to be somewhat like the dynamics of other two-player strategy games where the application of additional compute - thinking deeper into the game tree - confers an advantage. And so what is required to make defenders long-term successful might simply be to ensure they have more compute resources allocated to active, ongoing defense than attackers can muster. This is a strategy we could straightforwardly enact; national compute reserves allocated to cyber defense might be in order.