The AI systems in part I of the story are NOT “narrow” or “non-agentic”
There’s no difference between the level of “narrowness” or “agency” of the AI systems between parts I and II of the story.
Many people (including Richard Ngo and myself) seem to have interpreted part I as arguing that there could be an AI takeover by AI systems that are non-agentic and/or narrow (i.e. are not agentic AGI). But this is not at all what Paul intended to argue.
Put another way, both parts I and II are instances of the “second species” concern/gorilla problem: that AI systems will gain control of humanity’s future. (I think this is also identical to what people mean when they say “AI takeover”.)
As far as I can tell, this isn’t really a different kind of concern from the classic Bostrom-Yudkowsky case for AI x-risk. It’s just a more nuanced picture of what goes wrong, that also makes failure look plausible in slow takeoff worlds.
Instead, the key difference between parts I and II of the story is the way that the models’ objectives generalise.
In part II, it’s the kind of generalisation typically called a “treacherous turn”. The models learn the objective of “seeking influence”. Early in training, the best way to do that is by “playing nice”. The failure mode is that, once they become sufficiently capable, they no longer need to play nice and instead take control of humanity’s future.
In part I, it’s a different kind of generalisation, which has been much less discussed. The models learn some easily-measurable objective which isn’t what humans actually want. In other words, the failure mode is that these models are trying to “produce high scores” instead of “help humans get what they want”. You might think that using human feedback to specify the base objective will alleviate this problem (e.g. use learn a reward model from human demonstrations or preferences about a hard-to-measure objective). But this doesn’t obviously help: now, the failure mode is that the model learns the objective “do things that look to humans like you are achieving X” or “do things that the humans giving feedback about X will rate highly” (instead of “actually achieving X”).
Notice that in both of these scenarios, the models are mesa-optimizers (i.e. the learned models are themselves optimizers), and failure ensues because the models’ learned objectives generalise in the wrong way.
This was discussed in comments (on a separate post) by Richard Ngo and Paul Christiano. There’s a lot more important discussion in that comment thread, which is summarised in this doc.
The AI systems in part I of the story are NOT “narrow” or “non-agentic”
There’s no difference between the level of “narrowness” or “agency” of the AI systems between parts I and II of the story.
Many people (including Richard Ngo and myself) seem to have interpreted part I as arguing that there could be an AI takeover by AI systems that are non-agentic and/or narrow (i.e. are not agentic AGI). But this is not at all what Paul intended to argue.
Put another way, both parts I and II are instances of the “second species” concern/gorilla problem: that AI systems will gain control of humanity’s future. (I think this is also identical to what people mean when they say “AI takeover”.)
As far as I can tell, this isn’t really a different kind of concern from the classic Bostrom-Yudkowsky case for AI x-risk. It’s just a more nuanced picture of what goes wrong, that also makes failure look plausible in slow takeoff worlds.
Instead, the key difference between parts I and II of the story is the way that the models’ objectives generalise.
In part II, it’s the kind of generalisation typically called a “treacherous turn”. The models learn the objective of “seeking influence”. Early in training, the best way to do that is by “playing nice”. The failure mode is that, once they become sufficiently capable, they no longer need to play nice and instead take control of humanity’s future.
In part I, it’s a different kind of generalisation, which has been much less discussed. The models learn some easily-measurable objective which isn’t what humans actually want. In other words, the failure mode is that these models are trying to “produce high scores” instead of “help humans get what they want”. You might think that using human feedback to specify the base objective will alleviate this problem (e.g. use learn a reward model from human demonstrations or preferences about a hard-to-measure objective). But this doesn’t obviously help: now, the failure mode is that the model learns the objective “do things that look to humans like you are achieving X” or “do things that the humans giving feedback about X will rate highly” (instead of “actually achieving X”).
Notice that in both of these scenarios, the models are mesa-optimizers (i.e. the learned models are themselves optimizers), and failure ensues because the models’ learned objectives generalise in the wrong way.
This was discussed in comments (on a separate post) by Richard Ngo and Paul Christiano. There’s a lot more important discussion in that comment thread, which is summarised in this doc.