Demanding a bounded utility function seems like a horrible idea. For starters it violates our expectation of independence for causally isolated universes. In other words the utility of both universe A and B existing isn’t the sum of the utility of A existing and B existing.
Besides, intuitively if you were really offered with perfect confidence a bet with a 99.99...90% chance of winning you a dollar but if you lose all the souls in the world will burn in hell forever do you really think it would be a worthwhile bet to take?
What is bad about Pascal’s mugging isn’t the existence of arbitrarily bad outcomes, it’s the existence of arbitrarily bad expectations. What we really want is that the integral of utility relative to the probability measure is never infinite. This is a much more reasonable condition. In other words utility should be L1 integrabale with respect to your prior probability distribution (and as long as you never condition on events of probability 0 this property persists).
Besides, intuitively if you were really offered with perfect confidence a bet with a 99.99...90% chance of winning you a dollar but if you lose all the souls in the world will burn in hell forever do you really think it would be a worthwhile bet to take?
Just how many nines do those ellipses represent? That’s kind of important! I mean, if there is enough nines then the value of the dollar easily outweighs the risk and your intuition is deceiving you. Consider for example:
Given the information I currently have there is some non zero chance that I am wrong about the origin of the universe. It could be that there really is a God that will punish us with eternity in hell if we have the wrong belief about Him. A friendly superintelligence has a reasonable chance of figuring this out for us and telling us the Good News. Me having an extra dollar produces a non-zero increase in the probability that we create an AI. Multiplying non-zero chances together gives other nonzero chances. For any given probability of eternities in hell there is a number of nines such that (100 − 99.99#{n * “9”}90)% is a lower probability of eternities in hell.
Moral of this story: Be careful when throwing not-quite-infinities around to prove a point. Not-quite-infinities @#$% things up almost as much as infinities.
Besides, intuitively if you were really offered with perfect confidence a bet with a 99.99...90% chance of winning you a dollar but if you lose all the souls in the world will burn in hell forever do you really think it would be a worthwhile bet to take?
While I genuinely appreciate how implausible it is to consider that bet worthwhile, there is a flipside. See the The LIfespan Dilemma. The point is that people have inconsistent preferences here, so it is easy to find intuitive counterexamples to any position one could take. I find the bounded approach to be the least of all evils.
What is bad about Pascal’s mugging isn’t the existence of arbitrarily bad outcomes, it’s the existence of arbitrarily bad expectations. What we really want is that the integral of utility relative to the probability measure is never infinite.
It is actually pretty hard to achieve this. Consider a sequence of lives L1, L2, etc. such that (i) the temporary quality of each life increases over time in each life, and increases at the same rate, (ii) the duration of the lives in the sequence is increasing and approaching infinity, (iii) the utility of each lives in the sequence is approaching infinity. And now consider an infinitely long life L whose temporary quality increases at the same rate as L1, L2, etc. Plausibly, L is better than L1, L2, etc. This means L has a value which exceeds any finite value, and it means you will take any chance of L, no matter how small, to any of the Li, no matter how certain it would be. When choosing among different available actions which may lead to some of these lives, you would, in practice, consider only the chance of getting an infinitely long life, turning to other considerations only to break ties. Upshot: weak background assumptions + unbounded utility function --> obsessing over infinities, neglecting finite considerations except to break ties.
To be clear, the background assumptions are: that L, L1, L2, etc. are alternatives you could reasonably have non-zero probability in; that L is better than L1, L2, etc.; normal axioms of decision theory.
(You may also consider a variation of this argument involving histories of human civilization, rather than lives.)
Demanding a bounded utility function seems like a horrible idea. For starters it violates our expectation of independence for causally isolated universes. In other words the utility of both universe A and B existing isn’t the sum of the utility of A existing and B existing.
Besides, intuitively if you were really offered with perfect confidence a bet with a 99.99...90% chance of winning you a dollar but if you lose all the souls in the world will burn in hell forever do you really think it would be a worthwhile bet to take?
What is bad about Pascal’s mugging isn’t the existence of arbitrarily bad outcomes, it’s the existence of arbitrarily bad expectations. What we really want is that the integral of utility relative to the probability measure is never infinite. This is a much more reasonable condition. In other words utility should be L1 integrabale with respect to your prior probability distribution (and as long as you never condition on events of probability 0 this property persists).
Just how many nines do those ellipses represent? That’s kind of important! I mean, if there is enough nines then the value of the dollar easily outweighs the risk and your intuition is deceiving you. Consider for example:
Given the information I currently have there is some non zero chance that I am wrong about the origin of the universe. It could be that there really is a God that will punish us with eternity in hell if we have the wrong belief about Him. A friendly superintelligence has a reasonable chance of figuring this out for us and telling us the Good News. Me having an extra dollar produces a non-zero increase in the probability that we create an AI. Multiplying non-zero chances together gives other nonzero chances. For any given probability of eternities in hell there is a number of nines such that (100 − 99.99#{n * “9”}90)% is a lower probability of eternities in hell.
Moral of this story: Be careful when throwing not-quite-infinities around to prove a point. Not-quite-infinities @#$% things up almost as much as infinities.
While I genuinely appreciate how implausible it is to consider that bet worthwhile, there is a flipside. See the The LIfespan Dilemma. The point is that people have inconsistent preferences here, so it is easy to find intuitive counterexamples to any position one could take. I find the bounded approach to be the least of all evils.
It is actually pretty hard to achieve this. Consider a sequence of lives L1, L2, etc. such that (i) the temporary quality of each life increases over time in each life, and increases at the same rate, (ii) the duration of the lives in the sequence is increasing and approaching infinity, (iii) the utility of each lives in the sequence is approaching infinity. And now consider an infinitely long life L whose temporary quality increases at the same rate as L1, L2, etc. Plausibly, L is better than L1, L2, etc. This means L has a value which exceeds any finite value, and it means you will take any chance of L, no matter how small, to any of the Li, no matter how certain it would be. When choosing among different available actions which may lead to some of these lives, you would, in practice, consider only the chance of getting an infinitely long life, turning to other considerations only to break ties. Upshot: weak background assumptions + unbounded utility function --> obsessing over infinities, neglecting finite considerations except to break ties.
To be clear, the background assumptions are: that L, L1, L2, etc. are alternatives you could reasonably have non-zero probability in; that L is better than L1, L2, etc.; normal axioms of decision theory.
(You may also consider a variation of this argument involving histories of human civilization, rather than lives.)