At least from a traditional decision theory point of view, the point of assigning value to things is to rank them. In other words, the actual value of a thing is not relevant or even defined except in reference to other things. To say some outcome has “infinite value” merely suggests that this outcome outranks every other conceivable outcome. I’m not sure that designating something as having “infinite value” is a coherent application of the concept of value.
So, if you imagine (or discover) some outcome which you prefer more than any other conceivable outcome, you should always consider that you might’ve merely failed to conceive of a better alternative, meaning that your best alternative is merely your best known alternative, meaning you can’t guarantee that it is indeed the “best” alternative, meaning you shouldn’t jump the gun and assign it infinite value.
Saying something has infinite value is not the same as simply saying it outranks all other outcomes in your preference ordering. It is saying that a course of action that has a finite probability of producing it is superior to any course of action that does not have a finite probability of producing a comparable outcome, no matter how low that probability. (Provided, of course, that it does not come with a finite probability of an infinitely-bad outcome.)
In other words: utilities really do need to be treated as scalars, not just orderings, when reasoning under uncertainty.
If two courses of action differ in their probability of producing infinite utility, you might be able to make a coherent argument for maximizing the probability of infinite utility. But when you have multiple distinct infinite-utility outcomes, that gets harder. Really, anything nontrivially complex involving infinite utilities is hard.
At least from a traditional decision theory point of view, the point of assigning value to things is to rank them. In other words, the actual value of a thing is not relevant or even defined except in reference to other things. To say some outcome has “infinite value” merely suggests that this outcome outranks every other conceivable outcome. I’m not sure that designating something as having “infinite value” is a coherent application of the concept of value.
So, if you imagine (or discover) some outcome which you prefer more than any other conceivable outcome, you should always consider that you might’ve merely failed to conceive of a better alternative, meaning that your best alternative is merely your best known alternative, meaning you can’t guarantee that it is indeed the “best” alternative, meaning you shouldn’t jump the gun and assign it infinite value.
Saying something has infinite value is not the same as simply saying it outranks all other outcomes in your preference ordering. It is saying that a course of action that has a finite probability of producing it is superior to any course of action that does not have a finite probability of producing a comparable outcome, no matter how low that probability. (Provided, of course, that it does not come with a finite probability of an infinitely-bad outcome.)
In other words: utilities really do need to be treated as scalars, not just orderings, when reasoning under uncertainty.
If two courses of action differ in their probability of producing infinite utility, you might be able to make a coherent argument for maximizing the probability of infinite utility. But when you have multiple distinct infinite-utility outcomes, that gets harder. Really, anything nontrivially complex involving infinite utilities is hard.