You can’t reason with an agent which has a totally different utility function. The only reason that we can argue with humans is that they’re only human, and thus we share many desires.
If you know (or correctly guess) the agents’ utility function, and are able to communicate with it, then it may well be possible to reason with it.
Consider this situation; I am captured by a Paperclipper, which wishes to extract the iron from my blood and use it to make more paperclips (incidentally killing me in the process). I can attempt to escape by promising to send to the Paperclipper a quantity of iron—substantially more than can be found in my blood, and easier to extract—as soon as I am safe. As long as I can convince Clippy that I will follow through on my promise, I have a chance of living.
I can’t talk Clippy into adopting my own morality. But I can talk Clippy into performing individual actions that I would prefer Clippy to do (or into refraining from other actions) as long as I ensure that Clippy can get more paperclips by doing what I ask than by not doing what I ask.
Of course—my mistake. I meant that you can’t alter an agent’s desires by reason alone. You can’t appeal to desires you have. You can only appeal to its desires. So, when he’s going to turn the your blood iron into paperclips, and you want to live, you can’t try “But I want to live a long and happy life!”. If Clippy hasn’t got empathy, and you have nothing to offer that will help fulfill his own desires, then there’s nothing to be done, other than try to physical stop or kill him.
Maybe you’d be happier if you put him in a planet of his own, where a machine constantly destroye paperclips, and he was happy making new ones. My point is just that, if you do decide to make him happy, it’s not the optimal decision relative to a universal preference, or morality. It’s just the optimal decision relative to your desires. Is that ‘right’? Yes. That’s what we refer to, when we say ‘right’.
If you know (or correctly guess) the agents’ utility function, and are able to communicate with it, then it may well be possible to reason with it.
Consider this situation; I am captured by a Paperclipper, which wishes to extract the iron from my blood and use it to make more paperclips (incidentally killing me in the process). I can attempt to escape by promising to send to the Paperclipper a quantity of iron—substantially more than can be found in my blood, and easier to extract—as soon as I am safe. As long as I can convince Clippy that I will follow through on my promise, I have a chance of living.
I can’t talk Clippy into adopting my own morality. But I can talk Clippy into performing individual actions that I would prefer Clippy to do (or into refraining from other actions) as long as I ensure that Clippy can get more paperclips by doing what I ask than by not doing what I ask.
Of course—my mistake. I meant that you can’t alter an agent’s desires by reason alone. You can’t appeal to desires you have. You can only appeal to its desires. So, when he’s going to turn the your blood iron into paperclips, and you want to live, you can’t try “But I want to live a long and happy life!”. If Clippy hasn’t got empathy, and you have nothing to offer that will help fulfill his own desires, then there’s nothing to be done, other than try to physical stop or kill him.
Maybe you’d be happier if you put him in a planet of his own, where a machine constantly destroye paperclips, and he was happy making new ones. My point is just that, if you do decide to make him happy, it’s not the optimal decision relative to a universal preference, or morality. It’s just the optimal decision relative to your desires. Is that ‘right’? Yes. That’s what we refer to, when we say ‘right’.