The idea is that many different goals all have the useful subgoals of “acquire reasons to build stuff”, and humans/the-earth are made of atoms which are resources:
I don’t think that response makes sense. The classic instrumental convergence arguments are about a single agent; OP is asking why distinct AIs would coordinate with one another.
I think the AIs may well have goals that conflict with one another, just as humans’ goals do, but it’s plausible that they would form a coalition and work against humans’ interests because they expect a shared benefit, as humans sometimes do.
I agree with this, but also note that this topic is outside the scope of the post—it’s just about what would happen if AIs were aimed at defeating humanity, for whatever reason. It’s a separate question whether we should expect misaligned AIs to share enough goals, or have enough to gain from coordinating, to “team up.” I’ll say that if my main argument against catastrophe risk hinged on this (e.g., “We’re creating a bunch of AIs that would be able to defeat humanity if they coordinated, and would each individually like to defeat humanity, but won’t coordinate because of having different goals from eacha other”) I’d feel extremely nervous.
The idea is that many different goals all have the useful subgoals of “acquire reasons to build stuff”, and humans/the-earth are made of atoms which are resources:
https://arbital.com/p/instrumental_convergence/
I don’t think that response makes sense. The classic instrumental convergence arguments are about a single agent; OP is asking why distinct AIs would coordinate with one another.
I think the AIs may well have goals that conflict with one another, just as humans’ goals do, but it’s plausible that they would form a coalition and work against humans’ interests because they expect a shared benefit, as humans sometimes do.
I agree with this, but also note that this topic is outside the scope of the post—it’s just about what would happen if AIs were aimed at defeating humanity, for whatever reason. It’s a separate question whether we should expect misaligned AIs to share enough goals, or have enough to gain from coordinating, to “team up.” I’ll say that if my main argument against catastrophe risk hinged on this (e.g., “We’re creating a bunch of AIs that would be able to defeat humanity if they coordinated, and would each individually like to defeat humanity, but won’t coordinate because of having different goals from eacha other”) I’d feel extremely nervous.