Paper argues recursive AI self-improvement cannot yet be called safe

AIA paper argues that no one can call recursive self-improvement safe yet, because the field lacks a clear model of where it leads. Its author built a 2-player game showing the AI race is a Prisoner's Dilemma only at some risk levels, while at high risk mutual restraint can be a stable outcome.

Rohan Paul

@rohanpaul_ai

View on X

This paper argues that no one can call recursive self-improvement safe yet, because we lack even a clear way to think about where it leads.

Recursive self-improvement, or RSI, means an AI builds a better AI, which builds a better one, and so on. AI companies are starting to pursue it, with no shared view of when it would be safe.

As an ordinary user, the author got an AI to train a small LLM from scratch from 1 high-level prompt. At a http://FAR.AI safety workshop, experts listed very different conditions for RSI because they imagined very different outcomes.

His 2-player game shows the RSI race is a Prisoner's Dilemma only at some risk levels. At high risk, both sides holding back is a stable outcome, so waiting together can work.

Source details

Rohan Paul · x.comPublished Nov 10, 6:55 PM ET · added here 7:26 PM ET

Original title: This paper argues that no one can call recursive self-improvement safe yet, because we lack even a clear way to think about where it leads.

Paper argues recursive AI self-improvement cannot yet be called safe

Topics