Thank you for writing this reply. It definitely improved my overview of possible ways to look at this issue.
I guess your position can be summarized as “positive offense/defense balance will emerge soon, and aligned AI can block following unaligned AIs entirely if required”, is that roughly correct?
I have a few remarks about your ideas (not really a complete response).
The necessity for enforcing a ban even after AGI development is essentially entirely about failures of technical alignment.
First, in general, I think you’re underestimating the human component of alignment. Aligned AI should be aligned to something, namely humans. That means it won’t be able to build an industrial base in space until we’re ready to make it do that.
Even if we are not harmed by such a base in any way, and even if it would be legal to build it, I expect we may not be ready for it for a long time. It will be dead scary to see something develop that seems more powerful than us, but also deeply alien to us, even if tech companies insist it’s ‘aligned to our values’. Most people’s response will be to rein in its power, not expand it further. Any AI that’s aligned to us will need to take those feelings seriously.
Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it’s at all possible. And in any remotely democratic alignment plan, that’s a necessary step.
Second, I think it’s uncertain whether a level of AI that’s powerful enough to take over the world (and thereby cause existential risk) will also be powerful enough to build a large industrial base in space. If not, your plan might not work.
The biggest barrier to extreme regulatory measures like a ban is doubt (both reasonable and unreasonable) about the magnitude of misalignment risk.
I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don’t think alignment research will do that much here.
I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don’t think alignment research will do that much here.
I would say that the power of AI will continue to visibly massively expand (although underestimation of further developments will continue to be a big problem), but that will increase both ‘fear AI disaster’ and ‘get AI first’ elements. My read is that that the former is in a very difficult position now when its policy recommendations conflict with the latter. I see this in the Congressional hearings and rejection of the pause letter.
Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it’s at all possible. And in any remotely democratic alignment plan, that’s a necessary step.
When that kind of AI is available, it would mean by the same token that such expansion could break down MAD in short order as such explosive growth could give the power to safely disarm international rivals if not matched or stopped. And AI systems and developers will be able to demonstrate this. So the options would be verifying/trusting deals with geopolitical and ideological rivals to hold back or doing fast AI/industrial expansion. If dealmaking fails, then all options would look scary and abrupt.
Thank you for writing this reply. It definitely improved my overview of possible ways to look at this issue.
I guess your position can be summarized as “positive offense/defense balance will emerge soon, and aligned AI can block following unaligned AIs entirely if required”, is that roughly correct?
I have a few remarks about your ideas (not really a complete response).
First, in general, I think you’re underestimating the human component of alignment. Aligned AI should be aligned to something, namely humans. That means it won’t be able to build an industrial base in space until we’re ready to make it do that.
Even if we are not harmed by such a base in any way, and even if it would be legal to build it, I expect we may not be ready for it for a long time. It will be dead scary to see something develop that seems more powerful than us, but also deeply alien to us, even if tech companies insist it’s ‘aligned to our values’. Most people’s response will be to rein in its power, not expand it further. Any AI that’s aligned to us will need to take those feelings seriously.
Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it’s at all possible. And in any remotely democratic alignment plan, that’s a necessary step.
Second, I think it’s uncertain whether a level of AI that’s powerful enough to take over the world (and thereby cause existential risk) will also be powerful enough to build a large industrial base in space. If not, your plan might not work.
I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don’t think alignment research will do that much here.
I would say that the power of AI will continue to visibly massively expand (although underestimation of further developments will continue to be a big problem), but that will increase both ‘fear AI disaster’ and ‘get AI first’ elements. My read is that that the former is in a very difficult position now when its policy recommendations conflict with the latter. I see this in the Congressional hearings and rejection of the pause letter.
When that kind of AI is available, it would mean by the same token that such expansion could break down MAD in short order as such explosive growth could give the power to safely disarm international rivals if not matched or stopped. And AI systems and developers will be able to demonstrate this. So the options would be verifying/trusting deals with geopolitical and ideological rivals to hold back or doing fast AI/industrial expansion. If dealmaking fails, then all options would look scary and abrupt.