No, I’m asking which would be more right, and I don’t think it’s obvious.
If I like steak and you’re a vegetarian, and I’m feeling altruistic according to normal human standards, which do I give you? Steak or vegetables?
My utility function contains a term that sort of resembles a desire to increase your utility function, which is why I’d probably give you vegetables. But intuitively I feel like this only works when your utility function is close enough to mine that I can at least sympathize with it. I honestly don’t know what I’d do in the Pebble Sorter situation.
No, I’m asking which would be more right, and I don’t think it’s obvious.
If I like steak and you’re a vegetarian, and I’m feeling altruistic according to normal human standards, which do I give you? Steak or vegetables?
My utility function contains a term that sort of resembles a desire to increase your utility function, which is why I’d probably give you vegetables. But intuitively I feel like this only works when your utility function is close enough to mine that I can at least sympathize with it. I honestly don’t know what I’d do in the Pebble Sorter situation.
Well, if you want to do something that I would think was nice, then you give me whichever one I’d rather receive (presumably vegetables).
But are you concerned about what I would want, or what you would?
It’s time to go teach those baby-eating pebblesorters right from wrong!
(I don’t have any opinions about what to do for pebblesorters. Morality is about reality.)