To take it further, suppose that with 1% probability you are able to play a St. Petersburg game, and in the other 99% of worlds there is a billion years of torture. Then the story is that you don’t care about whether the probabilities are 1% and 99%, or 99% and 1%. Whether or not you find that unsatisfying is a personal call, but I find it extremely bad.
(But this proof doesn’t show that’s an inevitable consequence of Unbounded Utilities, it just shows that violating Dominance is an inevitable conclusion. So you might well think that this torture case is pretty unsatisfying but you can take or leave Dominance itself. I think that’s not crazy, but I think you’d be able to run a similar argument to get to any particular unsatisfying Dominance-violation.)
(I personally find violating Weak Dominance much more surprising, and that’s the point where I’m saying that you should just give up on talking about probabilistic mixtures. Though that may be too drastic. I’m phrasing this whole post in terms of dominance principles because I want to make the point that unbounded utilities basically force you to abandon very basic parts of your decision-theoretic machinery so you shouldn’t go on as if you have unbounded utilities but an otherwise normal decision theory.)
The point you mention about all decisions having infinite utility in expectation does seem worrying though—do you have an accessible intuition for why this is the case?
Basically just Pascal’s mugging. Under universal / non-dogmatic distributions, there is some probability on “Someone controls the universe, specifically searches for a series of outcomes with really large utility, and then runs the St. Petersburg game.”
(Of course for aggregative utilitarians you don’t even need to go there, any not-insane probability distribution over the size of the reachable universe is just obviously going to have infinite expectation.)
To take it further, suppose that with 1% probability you are able to play a St. Petersburg game, and in the other 99% of worlds there is a billion years of torture. Then the story is that you don’t care about whether the probabilities are 1% and 99%, or 99% and 1%. Whether or not you find that unsatisfying is a personal call, but I find it extremely bad.
(But this proof doesn’t show that’s an inevitable consequence of Unbounded Utilities, it just shows that violating Dominance is an inevitable conclusion. So you might well think that this torture case is pretty unsatisfying but you can take or leave Dominance itself. I think that’s not crazy, but I think you’d be able to run a similar argument to get to any particular unsatisfying Dominance-violation.)
(I personally find violating Weak Dominance much more surprising, and that’s the point where I’m saying that you should just give up on talking about probabilistic mixtures. Though that may be too drastic. I’m phrasing this whole post in terms of dominance principles because I want to make the point that unbounded utilities basically force you to abandon very basic parts of your decision-theoretic machinery so you shouldn’t go on as if you have unbounded utilities but an otherwise normal decision theory.)
Basically just Pascal’s mugging. Under universal / non-dogmatic distributions, there is some probability on “Someone controls the universe, specifically searches for a series of outcomes with really large utility, and then runs the St. Petersburg game.”
(Of course for aggregative utilitarians you don’t even need to go there, any not-insane probability distribution over the size of the reachable universe is just obviously going to have infinite expectation.)