I agree with this, but also note that this topic is outside the scope of the post—it’s just about what would happen if AIs were aimed at defeating humanity, for whatever reason. It’s a separate question whether we should expect misaligned AIs to share enough goals, or have enough to gain from coordinating, to “team up.” I’ll say that if my main argument against catastrophe risk hinged on this (e.g., “We’re creating a bunch of AIs that would be able to defeat humanity if they coordinated, and would each individually like to defeat humanity, but won’t coordinate because of having different goals from eacha other”) I’d feel extremely nervous.
I agree with this, but also note that this topic is outside the scope of the post—it’s just about what would happen if AIs were aimed at defeating humanity, for whatever reason. It’s a separate question whether we should expect misaligned AIs to share enough goals, or have enough to gain from coordinating, to “team up.” I’ll say that if my main argument against catastrophe risk hinged on this (e.g., “We’re creating a bunch of AIs that would be able to defeat humanity if they coordinated, and would each individually like to defeat humanity, but won’t coordinate because of having different goals from eacha other”) I’d feel extremely nervous.