From a Paul Christiano talk called “How Misalignment Could Lead to Takeover” (from February 2023):
Assume we’re in a world where AI systems are broadly deployed, and the world has become increasingly complex, where humans know less and less about how things work.
A viable strategy for AI takeover is to wait until there is certainty of success. If a ‘bad AI’ is smart, it will realize it won’t be successful if it tries to take over, not a problem.
So you lose when a takeover becomes possible, and some threshold of AIs behave badly. If all the smartest AIs you produce end up having the same problem, then takeover becomes more likely.
In other words, the smartest AIs will only take over if all the other AIs will also try to takeover.
Additionally, this is more likely to happen in an unstable world that could come from:
War (or other exogenous shocks): making it harder for humans to detect and respond to problems.
Cascading loss of control: some AIs try takeover and aren’t fully successful, but make the world increasingly crazy, which cascades into other AIs (who previously thought it would be unsuccessful) now joining the takeover.
Models interacting with each other: once some AIs realize could be successful, they may want to share that information with other models to see if other models may want to join in.
There’s this tempting feeling of training your AI against these takeover attempts (“hey, look at this bad AI who tried to takeover the data center, don’t do that!”), but you may just be training your model to learn that it needs to go much bigger if it wants to actually succeed at the takeover attempt.
Paul believes that if this kind of phenomenon is real, then he expects we can get compelling demonstrations in a lab (that would require some imagination to bridge the examples in the lab vs the wild). We’ll still get demonstrations in the wild, but unclear if they will be big enough to make humanity reconsider things.
From a Paul Christiano talk called “How Misalignment Could Lead to Takeover” (from February 2023):
Assume we’re in a world where AI systems are broadly deployed, and the world has become increasingly complex, where humans know less and less about how things work.
A viable strategy for AI takeover is to wait until there is certainty of success. If a ‘bad AI’ is smart, it will realize it won’t be successful if it tries to take over, not a problem.
So you lose when a takeover becomes possible, and some threshold of AIs behave badly. If all the smartest AIs you produce end up having the same problem, then takeover becomes more likely.
In other words, the smartest AIs will only take over if all the other AIs will also try to takeover.
Additionally, this is more likely to happen in an unstable world that could come from:
War (or other exogenous shocks): making it harder for humans to detect and respond to problems.
Cascading loss of control: some AIs try takeover and aren’t fully successful, but make the world increasingly crazy, which cascades into other AIs (who previously thought it would be unsuccessful) now joining the takeover.
Models interacting with each other: once some AIs realize could be successful, they may want to share that information with other models to see if other models may want to join in.
There’s this tempting feeling of training your AI against these takeover attempts (“hey, look at this bad AI who tried to takeover the data center, don’t do that!”), but you may just be training your model to learn that it needs to go much bigger if it wants to actually succeed at the takeover attempt.
Paul believes that if this kind of phenomenon is real, then he expects we can get compelling demonstrations in a lab (that would require some imagination to bridge the examples in the lab vs the wild). We’ll still get demonstrations in the wild, but unclear if they will be big enough to make humanity reconsider things.