I haven’t done the math, so take the following with a grain of salt.
We humans care about what will happen in the future. We care about how things will turn out. Call each possible future an “outcome”. We humans prefer some outcomes over another. We ought to steer the future towards the outcomes we prefer. Mathematically, we have a (perhaps partial) order on the set of outcomes, and if we had perfect knowledge of how our actions affected the future, our decision procedure would just be “pick the best outcome”.
So far I don’t think I’ve said anything controversial.
But we don’t have perfect knowledge of the future. We must reason and act under uncertainty. The best we can hope to do is assign conditional probabilities of to outcomes based on our possible actions. But in order to choose actions based on probabilities rather than certainties, we have to know a little more about what our preferences actually are. It’s not enough to know that one outcome is better than another, we have to know howmuch better. Let me give an example. If you are given the choice between winning a little money with probability .1 and a lot of money with probability .01, which option do you choose? Well, I haven’t given you enough information. If “a little” is $1 and “a lot” is $1 million, you should go for “a lot”. But if “a lot” is only $2, you’re better off going for “a little”.
So it turns out that if you want to have a consistent decision theory under these circumstances, there’s only one form it can take. Instead of a partial order on outcomes, you have to express your preference for each outcome as a number, called the utility of that outcome. And instead of selecting action that leads to the best outcome, you select the actions that lead to the highest expected utility.
The function that maps outcomes to utilities is called the utility function, and it is unique (given your preferences) up to an positive affine transformation. In other words, you can multiply the whole utility function by a positive scalar, or add any number to it, and it’s meaning does not change.
So you see, “maximize expected utility” doesn’t mean you have to maximize say, your profit, or your personal happiness, or even the number of human lives you save. All it really means is that your preferences ought to be consistent, because if they are, and if you’re trying the best you can to steer the future towards them, then “maximizing expected utility” is what you’re already doing.
I haven’t done the math, so take the following with a grain of salt.
We humans care about what will happen in the future. We care about how things will turn out. Call each possible future an “outcome”. We humans prefer some outcomes over another. We ought to steer the future towards the outcomes we prefer. Mathematically, we have a (perhaps partial) order on the set of outcomes, and if we had perfect knowledge of how our actions affected the future, our decision procedure would just be “pick the best outcome”.
So far I don’t think I’ve said anything controversial.
But we don’t have perfect knowledge of the future. We must reason and act under uncertainty. The best we can hope to do is assign conditional probabilities of to outcomes based on our possible actions. But in order to choose actions based on probabilities rather than certainties, we have to know a little more about what our preferences actually are. It’s not enough to know that one outcome is better than another, we have to know how much better. Let me give an example. If you are given the choice between winning a little money with probability .1 and a lot of money with probability .01, which option do you choose? Well, I haven’t given you enough information. If “a little” is $1 and “a lot” is $1 million, you should go for “a lot”. But if “a lot” is only $2, you’re better off going for “a little”.
So it turns out that if you want to have a consistent decision theory under these circumstances, there’s only one form it can take. Instead of a partial order on outcomes, you have to express your preference for each outcome as a number, called the utility of that outcome. And instead of selecting action that leads to the best outcome, you select the actions that lead to the highest expected utility.
The function that maps outcomes to utilities is called the utility function, and it is unique (given your preferences) up to an positive affine transformation. In other words, you can multiply the whole utility function by a positive scalar, or add any number to it, and it’s meaning does not change.
So you see, “maximize expected utility” doesn’t mean you have to maximize say, your profit, or your personal happiness, or even the number of human lives you save. All it really means is that your preferences ought to be consistent, because if they are, and if you’re trying the best you can to steer the future towards them, then “maximizing expected utility” is what you’re already doing.