Exactly! Bostrom seems to start the discussion from the point of humans having achieved a singleton as a species; in which case a conversation at this level would make more sense. But it seems that in order to operate as a unit, competing humans would have to work on the principle of a nuclear trigger where separate agents have to work in unison in order to launch. Thus we face the same problem with ourselves: how to know everyone in the keychain is honest? If the AI is anywhere near capable of taking control it may do so even partially and from there could wrangle the keys from the other players as needed. Competitive players are not likely to be cooperative unless they see some unfair advantage accruing to them in the future. (Why help the enemy advance unless we can see a way of gaining on them?) As long as we have human enemies, especially as our tools become increasingly powerful, the AI just needs to divide and conquer. Curses, foiled again!
Exactly! Bostrom seems to start the discussion from the point of humans having achieved a singleton as a species; in which case a conversation at this level would make more sense. But it seems that in order to operate as a unit, competing humans would have to work on the principle of a nuclear trigger where separate agents have to work in unison in order to launch. Thus we face the same problem with ourselves: how to know everyone in the keychain is honest? If the AI is anywhere near capable of taking control it may do so even partially and from there could wrangle the keys from the other players as needed. Competitive players are not likely to be cooperative unless they see some unfair advantage accruing to them in the future. (Why help the enemy advance unless we can see a way of gaining on them?) As long as we have human enemies, especially as our tools become increasingly powerful, the AI just needs to divide and conquer. Curses, foiled again!