I think the assumption that safe, aligned AI can’t defend against a later introduction of misaligned AI is false, or rather depends on the assumption of profound alignment failures so that the ‘aligned AI’ really isn’t. AI that is aligned enough to do AI research and operate industry and security forces can expand its capabilities to the technological frontier and grow an industrial base claiming unclaimed resources in space. Then any later AI introduced faces an insurmountable balance of capabilities just from the gap in resources, even if it catches up technologically. That would not violate the sovereignty of any state, although it could be seen as a violation of the Outer Space Treaty if not backed by the international community with treaty revision.
Advanced AI-enabled tech and industry can block bioweapons completely through physical barriers, detection, and sterilization. Vast wealth can find with high probability any zero-days that could be discovered with tiny wealth, and produce ultra-secure systems, so cyberattacks do not produce a vulnerable world. Even nuclear weapons lose their MAD element in the face of millions of drones/interceptors/defenses for each attempted attack (and humans can move to a distance in space, back up their minds, etc).
If it turns out there is something like the ability to create a vacuum collapse that enables one small actor to destroy a much larger AI-empowered civilization, then the vast civilization will find out first, and could safely enforce a ban if a negotiated AI-enforced treaty could not be struck.
If I understand correctly memes about pivotal acts to stop anyone from making misaligned AI stem from the view that we won’t be able to make AI that could be trusted to undergo intelligence explosion and industrial expansion for a long time after AI could enable some other ‘pivotal act.’ I.e. the necessity for enforcing a ban even after AGI development is essentially entirely about failures of technical alignment.
Furthermore, the biggest barrier to extreme regulatory measures like a ban is doubt (both reasonable and unreasonable) about the magnitude of misalignment risk, so research that studies and demonstrates high risk (if it is present) is perhaps the most leveraged possible tool to change the regulatory/governmental situation.
Thank you for writing this reply. It definitely improved my overview of possible ways to look at this issue.
I guess your position can be summarized as “positive offense/defense balance will emerge soon, and aligned AI can block following unaligned AIs entirely if required”, is that roughly correct?
I have a few remarks about your ideas (not really a complete response).
The necessity for enforcing a ban even after AGI development is essentially entirely about failures of technical alignment.
First, in general, I think you’re underestimating the human component of alignment. Aligned AI should be aligned to something, namely humans. That means it won’t be able to build an industrial base in space until we’re ready to make it do that.
Even if we are not harmed by such a base in any way, and even if it would be legal to build it, I expect we may not be ready for it for a long time. It will be dead scary to see something develop that seems more powerful than us, but also deeply alien to us, even if tech companies insist it’s ‘aligned to our values’. Most people’s response will be to rein in its power, not expand it further. Any AI that’s aligned to us will need to take those feelings seriously.
Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it’s at all possible. And in any remotely democratic alignment plan, that’s a necessary step.
Second, I think it’s uncertain whether a level of AI that’s powerful enough to take over the world (and thereby cause existential risk) will also be powerful enough to build a large industrial base in space. If not, your plan might not work.
The biggest barrier to extreme regulatory measures like a ban is doubt (both reasonable and unreasonable) about the magnitude of misalignment risk.
I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don’t think alignment research will do that much here.
I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don’t think alignment research will do that much here.
I would say that the power of AI will continue to visibly massively expand (although underestimation of further developments will continue to be a big problem), but that will increase both ‘fear AI disaster’ and ‘get AI first’ elements. My read is that that the former is in a very difficult position now when its policy recommendations conflict with the latter. I see this in the Congressional hearings and rejection of the pause letter.
Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it’s at all possible. And in any remotely democratic alignment plan, that’s a necessary step.
When that kind of AI is available, it would mean by the same token that such expansion could break down MAD in short order as such explosive growth could give the power to safely disarm international rivals if not matched or stopped. And AI systems and developers will be able to demonstrate this. So the options would be verifying/trusting deals with geopolitical and ideological rivals to hold back or doing fast AI/industrial expansion. If dealmaking fails, then all options would look scary and abrupt.
I think the assumption that safe, aligned AI can’t defend against a later introduction of misaligned AI is false, or rather depends on the assumption of profound alignment failures so that the ‘aligned AI’ really isn’t. AI that is aligned enough to do AI research and operate industry and security forces can expand its capabilities to the technological frontier and grow an industrial base claiming unclaimed resources in space. Then any later AI introduced faces an insurmountable balance of capabilities just from the gap in resources, even if it catches up technologically. That would not violate the sovereignty of any state, although it could be seen as a violation of the Outer Space Treaty if not backed by the international community with treaty revision.
Advanced AI-enabled tech and industry can block bioweapons completely through physical barriers, detection, and sterilization. Vast wealth can find with high probability any zero-days that could be discovered with tiny wealth, and produce ultra-secure systems, so cyberattacks do not produce a vulnerable world. Even nuclear weapons lose their MAD element in the face of millions of drones/interceptors/defenses for each attempted attack (and humans can move to a distance in space, back up their minds, etc).
If it turns out there is something like the ability to create a vacuum collapse that enables one small actor to destroy a much larger AI-empowered civilization, then the vast civilization will find out first, and could safely enforce a ban if a negotiated AI-enforced treaty could not be struck.
If I understand correctly memes about pivotal acts to stop anyone from making misaligned AI stem from the view that we won’t be able to make AI that could be trusted to undergo intelligence explosion and industrial expansion for a long time after AI could enable some other ‘pivotal act.’ I.e. the necessity for enforcing a ban even after AGI development is essentially entirely about failures of technical alignment.
Furthermore, the biggest barrier to extreme regulatory measures like a ban is doubt (both reasonable and unreasonable) about the magnitude of misalignment risk, so research that studies and demonstrates high risk (if it is present) is perhaps the most leveraged possible tool to change the regulatory/governmental situation.
Thank you for writing this reply. It definitely improved my overview of possible ways to look at this issue.
I guess your position can be summarized as “positive offense/defense balance will emerge soon, and aligned AI can block following unaligned AIs entirely if required”, is that roughly correct?
I have a few remarks about your ideas (not really a complete response).
First, in general, I think you’re underestimating the human component of alignment. Aligned AI should be aligned to something, namely humans. That means it won’t be able to build an industrial base in space until we’re ready to make it do that.
Even if we are not harmed by such a base in any way, and even if it would be legal to build it, I expect we may not be ready for it for a long time. It will be dead scary to see something develop that seems more powerful than us, but also deeply alien to us, even if tech companies insist it’s ‘aligned to our values’. Most people’s response will be to rein in its power, not expand it further. Any AI that’s aligned to us will need to take those feelings seriously.
Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it’s at all possible. And in any remotely democratic alignment plan, that’s a necessary step.
Second, I think it’s uncertain whether a level of AI that’s powerful enough to take over the world (and thereby cause existential risk) will also be powerful enough to build a large industrial base in space. If not, your plan might not work.
I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don’t think alignment research will do that much here.
I would say that the power of AI will continue to visibly massively expand (although underestimation of further developments will continue to be a big problem), but that will increase both ‘fear AI disaster’ and ‘get AI first’ elements. My read is that that the former is in a very difficult position now when its policy recommendations conflict with the latter. I see this in the Congressional hearings and rejection of the pause letter.
When that kind of AI is available, it would mean by the same token that such expansion could break down MAD in short order as such explosive growth could give the power to safely disarm international rivals if not matched or stopped. And AI systems and developers will be able to demonstrate this. So the options would be verifying/trusting deals with geopolitical and ideological rivals to hold back or doing fast AI/industrial expansion. If dealmaking fails, then all options would look scary and abrupt.
Your first three paragraphs are a profound and hopeful message and a reminder of the stakes