An AI can indeed have preferences that conflict with human preferences, but if it doesn’t start out with such preferences, it’s unclear how it comes to have them later.
We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted. For example, babies do not start out preferring Bach to Beethoven or Beethoven to Bach, but adults are able to develop that preference, even if it is not clear at this point how they come to do so.
If you could do so easily and with complete impunity, would you organize fights to death for your pleasure?
Voters have the ability to vote for policies and to do so easily and with complete impunity (nobody retaliates against a voter for his vote). And, unsurprisingly, voters regularly vote to take from others to give unto themselves—which is something they would never do in person (unless they were criminals, such as muggers or burglars). Moreover humans have an awe-inspiring capacity to clothe their rapaciousness in fine-sounding rhetoric.
Moreover, humans are often tempted to do things they know they shouldn’t, because they also have selfish desires. AIs don’t if you don’t build it into them.
Conflict does not require selfish desires. Any desire, of whatever sort, could potentially come into conflict with another person’s desire, and when there are many minds each with its own set of desires then conflict is almost inevitable. So the problem does not, in fact, turn on whether the mind is “selfish” or not. Any sort of desire can create the conflict, and conflict as such creates the problem I described. In a nutshell: evil men need not be selfish. A man such as Pol Pot could indeed have wanted nothing for himself and still ended up murdering millions of his countrymen.
An AI can indeed have preferences that conflict with human preferences, but if it doesn’t start out with such preferences, it’s unclear how it comes to have them later.
We do not know very well how the human mind does anything at all. But that the the human mind comes to have preferences that it did not have initially, cannot be doubted. For example, babies do not start out preferring Bach to Beethoven or Beethoven to Bach, but adults are able to develop that preference, even if it is not clear at this point how they come to do so.
If you could do so easily and with complete impunity, would you organize fights to death for your pleasure?
Voters have the ability to vote for policies and to do so easily and with complete impunity (nobody retaliates against a voter for his vote). And, unsurprisingly, voters regularly vote to take from others to give unto themselves—which is something they would never do in person (unless they were criminals, such as muggers or burglars). Moreover humans have an awe-inspiring capacity to clothe their rapaciousness in fine-sounding rhetoric.
Moreover, humans are often tempted to do things they know they shouldn’t, because they also have selfish desires. AIs don’t if you don’t build it into them.
Conflict does not require selfish desires. Any desire, of whatever sort, could potentially come into conflict with another person’s desire, and when there are many minds each with its own set of desires then conflict is almost inevitable. So the problem does not, in fact, turn on whether the mind is “selfish” or not. Any sort of desire can create the conflict, and conflict as such creates the problem I described. In a nutshell: evil men need not be selfish. A man such as Pol Pot could indeed have wanted nothing for himself and still ended up murdering millions of his countrymen.