What happens if your preferences do not satisfy Continuity? Say, you want to save human lives, but you’re not willing to incur any probability, no matter how small, of infinitely many people getting tortured infinitely long for this?
Then you basically have a two-step optimization; “find me the set of actions that have a minimal number of infinitely many people getting tortured infinitely long, and then of that set, find me the set of actions that save a maximal number of human lives.” The trouble with that is that people like to express their preferences with rules like that, but those preferences are not ones that they reflectively endorse. For example, would you rather it be certain that all intelligent life in the universe is destroyed forever, or there be a one out of R chance that infinitely many people get tortured for an infinitely long period, and R-1 out of R chance that humanity continues along happily? If R is sufficiently large (say, x^x with x ^s, with x equal to the number of atoms in the universe), then it seems that the first option is obviously worse.
A way to think about this is that infinities destroy averages, and VNM relies on scoring actions by their average utility. If utilities are bounded, then Continuity holds, and average utility always gives the results you expect if you measured utility correctly.
Then you basically have a two-step optimization; “find me the set of actions that have a minimal number of infinitely many people getting tortured infinitely long, and then of that set, find me the set of actions that save a maximal number of human lives.” The trouble with that is that people like to express their preferences with rules like that, but those preferences are not ones that they reflectively endorse. For example, would you rather it be certain that all intelligent life in the universe is destroyed forever, or there be a one out of R chance that infinitely many people get tortured for an infinitely long period, and R-1 out of R chance that humanity continues along happily? If R is sufficiently large (say, x^x with x ^s, with x equal to the number of atoms in the universe), then it seems that the first option is obviously worse.
A way to think about this is that infinities destroy averages, and VNM relies on scoring actions by their average utility. If utilities are bounded, then Continuity holds, and average utility always gives the results you expect if you measured utility correctly.