I think #3 and #4 are issues, but can be compensated for if aligned AIs outnumber or outclass misaligned AIs by enough. The situation seems fairly analogous to how things are with humans—law-abiding people face a lot of extra constraints, but are still collectively more powerful.
I think #1 is a risk, but it seems <<50% likely to be decisive, especially when considering (a) the possibility for things like space travel, hardened refuges, intense medical interventions, digital people, etc. that could become viable with aligned AIs; (b) the possibility that a relatively small number of biological humans’ surviving could still be enough to stop misaligned AIs (if we posit that aligned AIs greatly outnumber misaligned AIs). And I think misaligned AIs are less likely to cause any damage if the odds are against ultimately achieving their aims.
I also suspect that the disagreement on point #1 is infecting #2 and #4 a bit—you seem to be picturing scenarios where a small number of misaligned AIs can pose threats that can *only* be defended against with extremely intense, scary, sudden measures.
I’m pretty not sold on #2. There are stories like this you can tell, but I think there could be significant forces pushing the other way, such as a desire not to fall behind others’ capabilities. In a world where there are lots of powerful AIs and they’re continually advancing, I think the situation looks less like “Here’s a singular terrifying AI for you to integrate into your systems” and more like “Here’s the latest security upgrade, I think you’re getting pwned if you skip it.”
Finally, you seem to have focused heavily here on the “defense/deterrence/hardening” part of the picture, which I think *might* be sufficient, but isn’t the only tool in the toolkit. Many of the other AI uses in that section are about stopping misaligned AIs from being developed and deployed in the first place, which could make it much easier for them to be radically outnumbered/outclassed.
(Chiming in late, sorry!)
I think #3 and #4 are issues, but can be compensated for if aligned AIs outnumber or outclass misaligned AIs by enough. The situation seems fairly analogous to how things are with humans—law-abiding people face a lot of extra constraints, but are still collectively more powerful.
I think #1 is a risk, but it seems <<50% likely to be decisive, especially when considering (a) the possibility for things like space travel, hardened refuges, intense medical interventions, digital people, etc. that could become viable with aligned AIs; (b) the possibility that a relatively small number of biological humans’ surviving could still be enough to stop misaligned AIs (if we posit that aligned AIs greatly outnumber misaligned AIs). And I think misaligned AIs are less likely to cause any damage if the odds are against ultimately achieving their aims.
I also suspect that the disagreement on point #1 is infecting #2 and #4 a bit—you seem to be picturing scenarios where a small number of misaligned AIs can pose threats that can *only* be defended against with extremely intense, scary, sudden measures.
I’m pretty not sold on #2. There are stories like this you can tell, but I think there could be significant forces pushing the other way, such as a desire not to fall behind others’ capabilities. In a world where there are lots of powerful AIs and they’re continually advancing, I think the situation looks less like “Here’s a singular terrifying AI for you to integrate into your systems” and more like “Here’s the latest security upgrade, I think you’re getting pwned if you skip it.”
Finally, you seem to have focused heavily here on the “defense/deterrence/hardening” part of the picture, which I think *might* be sufficient, but isn’t the only tool in the toolkit. Many of the other AI uses in that section are about stopping misaligned AIs from being developed and deployed in the first place, which could make it much easier for them to be radically outnumbered/outclassed.