What solution do people prefer to Pascal’s Mugging? I know of three approaches:
1) Handing over the money is the right thing to do exactly as the calculation might indicate.
2) Debiasing against overconfidence shouldn’t mean having any confidence in what others believe, but just reducing our own confidence; thus the expected gain if we’re wrong is found by drawing from a broader reference class, like “offers from a stranger”.
3) The calculation is correct, but we must pre-commit to not paying under such circumstances in order not to be gamed.
The unbounded utility function (in some physical objects that can be tiled indefinitely) in Pascal’s mugging gives infinite expected utility to all actions, and no reason to prefer handing over the money to any other action. People don’t actually show the pattern of preferences implied by an unbounded utility function.
If we make the utility function a bounded function of happy lives (or other tilable physical structures) with a high bound, other possibilities will offer high expected utility. The Mugger is not the most credible way to get huge rewards (investing in our civilization on the chance that physics allows unlimited computation beats the Mugger). This will be the case no matter how huge we make the (finite) bound.
Bounding the utility function definitely solves the problem, but there are a couple of problems. One is the principle that the utility function is not up for grabs, the other is that a bounded utility function has some rather nasty consequences of the “leave one baby on the track” kind.
One is the principle that the utility function is not up for grabs,
I don’t buy this. Many people have inconsistent intuitions regarding aggregation, as with population ethics. Someone with such inconsistent preferences doesn’t have a utility function to preserve.
Also note that a bounded utility function can allot some of the potential utility under the bound to producing an infinite amount of stuff, and that as a matter of psychological fact the human emotional response to stimuli can’t scale indefinitely with bigger numbers.
And, of course, allowing unbounded growth of utility with some tilable physical process means that process can dominate the utility of any non-aggregative goods, e.g. the existence of at least some instantiations of art or knowledge, or overall properties of the world like ratios of very good to lives just barely worth living/creating (although you might claim that the value of the last scales with population size, many wouldn’t characterize it that way).
Bounded utility functions seem to come much closer to letting you represent actual human concerns, or to represent more of them, in my view.
Eliezer’s original article bases its argument on the use of Solomonoff induction. He even suggests up front what the problem with it is, although the comments don’t make anything of it: SI is based solely on program length and ignores computational resources. The optimality theorems around SI depend on the same assumption. Therefore I suggest:
4. Pascal’s Mugging is a refutation of the Solomonoff prior.
But where a computationally bounded agent, or an unbounded one that cares how much work it does, should get its priors from instead would require more thought than a few minutes on a lunchtime break.
In one sense you can’t use evidence to argue with a prior, but I think that factoring in computational resources as a cost would have put you on the wrong side of a lot of our discoveries about the Universe.
In one sense you can’t use evidence to argue with a prior, but I think that factoring in computational resources as a cost would have put you on the wrong side of a lot of our discoveries about the Universe.
Could you expand that with examples? And if you can’t use evidence to argue with a prior, what can you use?
I’m thinking of the way we keep finding ways in which the Universe is far larger than we’d imagined—up to and including the quantum multiverse, and possibly one day including a multiverse-based solution to the fine tuning problem.
The whole point about a prior is that it’s where you start before you’ve seen the evidence. But in practice using evidence to choose a prior is likely justified on the grounds that our actual prior is whatever we evolved with or whatever evolution’s implicit prior is, and settling on a formal prior with which to attack hard problems is something we do in the face of lots of evidence. I think.
I’m thinking of the way we keep finding ways in which the Universe is far larger than we’d imagined
It’s not clear to me how that bears on the matter. I would need to see something with some mathematics in it.
The whole point about a prior is that it’s where you start before you’ve seen the evidence.
There’s a potential infinite regress if you argue that changing your prior on seeing the evidence means it was never your prior, but something prior to it was.
You can go on questioning those previous priors, and so on indefinitely, and therefore nothing is really a prior.
You stop somewhere with an unquestionable prior, and the only unquestionable truths are those of mathematics, therefore there is an Original Prior that can be deduced by pure thought. (Calvinist Bayesianism, one might call it. No agent has the power to choose its priors, for it would have to base its choice on something prior to those priors. Nor can it priors be conditional in any way upon any property of that agent, for then again they would not be prior. The true Prior is prior to all things, and must therefore be inherent in the mathematical structure of being. This Prior is common to all agents but in their fundamentally posterior state they are incapable of perceiving it. I’m tempted to pastiche the whole Five Points of Calvinism, but that’s enough for the moment.)
You stop somewhere, because life is short, with a prior that appears satisfactory for the moment, but which one allows the possibility of later rejecting.
I think 1 and 2 are non-starters, and 3 allows for evidence defeating priors.
Tom_McCabe2 suggests generalizing EY’s rebuttal of Pascal’s Wager to Pascal’s Mugging: it’s not actually obvious that someone claiming they’ll destroy 3^^^^3 people makes it more likely that 3^^^^3 people will die. The claim is arguably such weak evidence that it’s still about equally likely that handing over the $5 will kill 3^^^^3 people, and if the two probabilities are sufficiently equal, they’ll cancel out enough to make it not worth handing over the $5.
Personally, I always just figured that the probability of someone (a) threatening me with killing 3^^^^3 people, (b) having the ability to do so, and (c) not going ahead and killing the people anyway after I give them the $5, is going to be way less than 1/3^^^^3, so the expected utility of giving the mugger the $5 is almost certainly less than the $5 of utility I get by hanging on to it. In which case there is no problem to fix. EY claims that the Solomonoff-calculated probability of someone having ‘magic powers from outside the Matrix’ ‘isn’t anywhere near as small as 3^^^^3 is large,’ but to me that just suggests that the Solomonoff calculation is too credulous.
(Edited to try and improve paraphrase of Tom_McCabe2.)
This seems very similar to the “reference class fallback” approach to confidence set out in point 2, but I prefer to explicitly refer to reference classes when setting out that approach, otherwise the exactly even odds you apply to massively positive and massively negative utility here seem to come rather conveniently out of a hat...
Fair enough. Actually, looking at my comment again, I think I paraphrased Tom_McCabe2 really badly, so thanks for replying and making me take another look! I’ll try and edit my comment so it’s a better paraphrase.
I’m not sure this problem needs a “solution” in the sense that everyone here seems to accept. Human beings have preferences. Utility functions are an imperfect way of modeling those preferences, not some paragon of virtue that everyone should aspire to. Most models break down when pushed outside their area of applicability.
The utility function assumes that you play the “game” (situation, whatever) an infinite number of times and then find the net utility. Thats good when your playing the “game” enough times to matter. It’s not when your only playing a small number of times. So lets look at it as “winning” or “loosing”. If the odds are really low and the risk is high and your only playing once, then most of the time you expect to loose. If you do it enough times, you even the odds out and the loss gets canceled out by the large reward, but only playing once you expect to loose more then you gain. Why would you assume differnetly? Thats my 2 cents and so far its the only way I have come up with to navigate around this problem.
The utility function assumes that you play the “game” (situation, whatever) an infinite number of times and then find the net utility.
This isn’t right. The way utility is normally defined, if outcome X has 10 times the utility of outcome Y for a given utility function, agents behaving in accord with that function will be indifferent between certain Y and a 10% probability of X. That’s why they call expected utility theory a theory of “decision under uncertainty.” The scenario you describe sounds like one where the payoffs are in some currency such that you have declining utility with increasing amounts of the currency.
The scenario you describe sounds like one where the payoffs are in some currency such that you have declining utility with increasing amounts of the currency.
Uh, no. Allright, lets say I give you a 1 out of 10 chance at winning 10 times everything you own, but the other 9 times you lose everything. The net utility for accepting is the same as not accepting, yet thats completely ignoring the fact that if you do enter, 90 % of the time you lose everything, no matter how high the reward is.
As Thom indicates, this is exactly what I was talking about: ten times the stuff you own, rather than ten times the utility. Since utility is just a representation of your preferences, the 1 in 10 payoff would only have ten times the utility of your current endowment if you would be willing to accept this gamble.
That’s only true if “everything you own” is cast in terms of utility, which is not intuitive. Normally, “everything you own” would be in terms of dollars or something to that effect, and ten times the number of dollars I have is not worth 10 times the utility of those dollars.
What solution do people prefer to Pascal’s Mugging? I know of three approaches:
1) Handing over the money is the right thing to do exactly as the calculation might indicate.
2) Debiasing against overconfidence shouldn’t mean having any confidence in what others believe, but just reducing our own confidence; thus the expected gain if we’re wrong is found by drawing from a broader reference class, like “offers from a stranger”.
3) The calculation is correct, but we must pre-commit to not paying under such circumstances in order not to be gamed.
What have I left out?
The unbounded utility function (in some physical objects that can be tiled indefinitely) in Pascal’s mugging gives infinite expected utility to all actions, and no reason to prefer handing over the money to any other action. People don’t actually show the pattern of preferences implied by an unbounded utility function.
If we make the utility function a bounded function of happy lives (or other tilable physical structures) with a high bound, other possibilities will offer high expected utility. The Mugger is not the most credible way to get huge rewards (investing in our civilization on the chance that physics allows unlimited computation beats the Mugger). This will be the case no matter how huge we make the (finite) bound.
Bounding the utility function definitely solves the problem, but there are a couple of problems. One is the principle that the utility function is not up for grabs, the other is that a bounded utility function has some rather nasty consequences of the “leave one baby on the track” kind.
I don’t buy this. Many people have inconsistent intuitions regarding aggregation, as with population ethics. Someone with such inconsistent preferences doesn’t have a utility function to preserve.
Also note that a bounded utility function can allot some of the potential utility under the bound to producing an infinite amount of stuff, and that as a matter of psychological fact the human emotional response to stimuli can’t scale indefinitely with bigger numbers.
And, of course, allowing unbounded growth of utility with some tilable physical process means that process can dominate the utility of any non-aggregative goods, e.g. the existence of at least some instantiations of art or knowledge, or overall properties of the world like ratios of very good to lives just barely worth living/creating (although you might claim that the value of the last scales with population size, many wouldn’t characterize it that way).
Bounded utility functions seem to come much closer to letting you represent actual human concerns, or to represent more of them, in my view.
Eliezer’s original article bases its argument on the use of Solomonoff induction. He even suggests up front what the problem with it is, although the comments don’t make anything of it: SI is based solely on program length and ignores computational resources. The optimality theorems around SI depend on the same assumption. Therefore I suggest:
4. Pascal’s Mugging is a refutation of the Solomonoff prior.
But where a computationally bounded agent, or an unbounded one that cares how much work it does, should get its priors from instead would require more thought than a few minutes on a lunchtime break.
In one sense you can’t use evidence to argue with a prior, but I think that factoring in computational resources as a cost would have put you on the wrong side of a lot of our discoveries about the Universe.
Could you expand that with examples? And if you can’t use evidence to argue with a prior, what can you use?
I’m thinking of the way we keep finding ways in which the Universe is far larger than we’d imagined—up to and including the quantum multiverse, and possibly one day including a multiverse-based solution to the fine tuning problem.
The whole point about a prior is that it’s where you start before you’ve seen the evidence. But in practice using evidence to choose a prior is likely justified on the grounds that our actual prior is whatever we evolved with or whatever evolution’s implicit prior is, and settling on a formal prior with which to attack hard problems is something we do in the face of lots of evidence. I think.
It’s not clear to me how that bears on the matter. I would need to see something with some mathematics in it.
There’s a potential infinite regress if you argue that changing your prior on seeing the evidence means it was never your prior, but something prior to it was.
You can go on questioning those previous priors, and so on indefinitely, and therefore nothing is really a prior.
You stop somewhere with an unquestionable prior, and the only unquestionable truths are those of mathematics, therefore there is an Original Prior that can be deduced by pure thought. (Calvinist Bayesianism, one might call it. No agent has the power to choose its priors, for it would have to base its choice on something prior to those priors. Nor can it priors be conditional in any way upon any property of that agent, for then again they would not be prior. The true Prior is prior to all things, and must therefore be inherent in the mathematical structure of being. This Prior is common to all agents but in their fundamentally posterior state they are incapable of perceiving it. I’m tempted to pastiche the whole Five Points of Calvinism, but that’s enough for the moment.)
You stop somewhere, because life is short, with a prior that appears satisfactory for the moment, but which one allows the possibility of later rejecting.
I think 1 and 2 are non-starters, and 3 allows for evidence defeating priors.
What do you mean by “evolution’s implicit prior”?
Tom_McCabe2 suggests generalizing EY’s rebuttal of Pascal’s Wager to Pascal’s Mugging: it’s not actually obvious that someone claiming they’ll destroy 3^^^^3 people makes it more likely that 3^^^^3 people will die. The claim is arguably such weak evidence that it’s still about equally likely that handing over the $5 will kill 3^^^^3 people, and if the two probabilities are sufficiently equal, they’ll cancel out enough to make it not worth handing over the $5.
Personally, I always just figured that the probability of someone (a) threatening me with killing 3^^^^3 people, (b) having the ability to do so, and (c) not going ahead and killing the people anyway after I give them the $5, is going to be way less than 1/3^^^^3, so the expected utility of giving the mugger the $5 is almost certainly less than the $5 of utility I get by hanging on to it. In which case there is no problem to fix. EY claims that the Solomonoff-calculated probability of someone having ‘magic powers from outside the Matrix’ ‘isn’t anywhere near as small as 3^^^^3 is large,’ but to me that just suggests that the Solomonoff calculation is too credulous.
(Edited to try and improve paraphrase of Tom_McCabe2.)
This seems very similar to the “reference class fallback” approach to confidence set out in point 2, but I prefer to explicitly refer to reference classes when setting out that approach, otherwise the exactly even odds you apply to massively positive and massively negative utility here seem to come rather conveniently out of a hat...
Fair enough. Actually, looking at my comment again, I think I paraphrased Tom_McCabe2 really badly, so thanks for replying and making me take another look! I’ll try and edit my comment so it’s a better paraphrase.
I’m not sure this problem needs a “solution” in the sense that everyone here seems to accept. Human beings have preferences. Utility functions are an imperfect way of modeling those preferences, not some paragon of virtue that everyone should aspire to. Most models break down when pushed outside their area of applicability.
The utility function assumes that you play the “game” (situation, whatever) an infinite number of times and then find the net utility. Thats good when your playing the “game” enough times to matter. It’s not when your only playing a small number of times. So lets look at it as “winning” or “loosing”. If the odds are really low and the risk is high and your only playing once, then most of the time you expect to loose. If you do it enough times, you even the odds out and the loss gets canceled out by the large reward, but only playing once you expect to loose more then you gain. Why would you assume differnetly? Thats my 2 cents and so far its the only way I have come up with to navigate around this problem.
This isn’t right. The way utility is normally defined, if outcome X has 10 times the utility of outcome Y for a given utility function, agents behaving in accord with that function will be indifferent between certain Y and a 10% probability of X. That’s why they call expected utility theory a theory of “decision under uncertainty.” The scenario you describe sounds like one where the payoffs are in some currency such that you have declining utility with increasing amounts of the currency.
Uh, no. Allright, lets say I give you a 1 out of 10 chance at winning 10 times everything you own, but the other 9 times you lose everything. The net utility for accepting is the same as not accepting, yet thats completely ignoring the fact that if you do enter, 90 % of the time you lose everything, no matter how high the reward is.
As Thom indicates, this is exactly what I was talking about: ten times the stuff you own, rather than ten times the utility. Since utility is just a representation of your preferences, the 1 in 10 payoff would only have ten times the utility of your current endowment if you would be willing to accept this gamble.
That’s only true if “everything you own” is cast in terms of utility, which is not intuitive. Normally, “everything you own” would be in terms of dollars or something to that effect, and ten times the number of dollars I have is not worth 10 times the utility of those dollars.