The obvious problem with this is that your utility is not defined if you are willing to accept muggings, so you can’t use the framework of expected utility maximization at all. The point of the mugger is just to illustrate this, I don’t think anyone thinks you should actually pay them (after all, you might encounter a more generous mugger tomorrow, or any number of more realistic opportunities to do astronomical amounts of good...)
Part of the issue is that I am coming at this problem from a different perspective than maybe you or Eliezer is. I believe that paying the mugger is basically worthless in the sense that doing almost any old good thing is better than paying the mugger. I would like to have a satisfying explanation of this. In contrast, Eliezer is interested in reconciling a view about complexity priors with a view about utility functions, and the mugger is an illustration of the conflict.
I do not have a proposed reconciliation of complexity priors and unbounded utility functions. Instead, the above comment is a recommended as an explanation of why paying the mugger is basically worthless in comparison with ordinary things you could do. So this hypothesis would say that if you set up your priors and your utility function in a reasonable way, the expected utility of downstream effects of ordinary good actions would greatly exceed the expected utility of paying the mugger.
Even if you decided that the expected utility framework somehow breaks down in cases like this, I think various related claims would still be plausible. E.g., rather than saying that doing ordinary good things has higher expected utility, it would be plausible that doing ordinary good things is “better relative to your uncertainty” than paying the mugger.
On a different note, another thing I find unsatisfying about the downstream effects reply is that it doesn’t seem to match up with why ordinary people think it is dumb to pay the mugger. The ultimate reason I think it is dumb to pay the mugger is strongly related to why ordinary people think it is dumb to pay the mugger, and I would like to be able to thoroughly understand the most plausible common-sense explanation of why paying the mugger is dumb. The proposed relationship between ordinary actions and their distant effects seems too far off from why common sense would say that paying the mugger is dumb. I guess this is ultimately pretty close to one of Nick Bostrom’s complaints about empirical stabilizing assumptions.
I believe that paying the mugger is basically worthless in the sense that doing almost any old good thing is better than paying the mugger.
I think we are all in agreement with this (modulo the fact that all of the expected values end up being infinite and so we can’t compare in the normal way; if you e.g. proposed a cap of 3^^^^^^^3 on utilities, then you certainly wouldn’t pay the mugger).
On a different note, another thing I find unsatisfying about the downstream effects reply is that it doesn’t seem to match up with why ordinary people think it is dumb to pay the mugger.
It seems very likely to me that ordinary people are best modeled as having bounded utility functions, which would explain the puzzle.
So it seems like there are two issues:
You would never pay the mugger in any case, because other actions are better.
If you object to the fact that the only thing you care about is a very small probability of an incredibly good outcome, then that’s basically the definition of having a bounded utility function.
And then there is the third issue Eliezer is dealing with, where he wants to be able to have an unbounded utility function even if that doesn’t describe anyone’s preferences (since it seems like boundedness is an unfortunate restriction to randomly impose on your preferences for technical reasons), and formally it’s not clear how to do that. At the end of the post he seems to suggest giving up on that though.
Obviously to really put the idea of people having bounded utility functions to the test, you have to forget about it solving problems of small probabilities and incredibly good outcomes and focus on the most unintuitive consequences of it. For one, having a bounded utility function means caring arbitrarily little about differences between the goodness of different sufficiently good outcomes. And all the outcomes could be certain too. You could come up with all kinds of thought experiments involving purchasing huge numbers of years happy life or some other good for a few cents. You know all of this so I wonder why you don’t talk about it.
Also I believe that Eliezer thinks that an unbounded utility function describes at least his preferences. I remember he made a comment about caring about new happy years of life no matter how many he’d already been granted.
(I haven’t read most of the discussion in this thread or might just be missing something so this might be irrelevant.)
As far as I know the strongest version of this argument is Benja’s, here (which incidentally seems to deserve many more upvotes than it got).
Benja’s scenario isn’t a problem for normal people though, who are not reflectively consistent and whose preferences manifestly change over time.
Beyond that, it seems like people’s preferences regarding the lifespan dilemma are somewhat confusing and probably inconsistent, much like their preferences regarding the repugnant conclusion. But that seems mostly orthogonal to pascal’s mugging, and the basic point—having unbounded utility by definition means you are willing to accept negligible chances of sufficiently good outcomes against probability nearly 1 of any fixed bad outcome, so if you object to the latter you are just objecting to unbounded utility.
I agree I was being uncharitable towards Eliezer. But it is true that at the end of this post he was suggesting giving up on unbounded utility, and that everyone in this crowd seems to ultimately take that route.
I think we are all in agreement with this (modulo the fact that all of the expected values end up being infinite and so we can’t compare in the normal way; if you e.g. proposed a cap of 3^^^^^^^3 on utilities, then you certainly wouldn’t pay the mugger).
Sorry, I didn’t mean to suggest otherwise. The “different perspective” part was supposed to be about the “in contrast” part.
It seems very likely to me that ordinary people are best modeled as having bounded utility functions, which would explain the puzzle.
I agree with yli that this has other unfortunate consequences. And, like Holden, I find it unfortunate to have to say that saving N lives with probability 1/N is worse than saving 1 life with probability 1. I also recognize that the things I would like to say about this collection of cases are inconsistent with each other. It’s a puzzle. I have written about this puzzle at reasonable length in my dissertation. I tend to think that bounded utility functions are the best consistent solution I know of, but that continuing to operate with inconsistent preferences (in a tasteful way) may be better in practice.
The obvious problem with this is that your utility is not defined if you are willing to accept muggings, so you can’t use the framework of expected utility maximization at all. The point of the mugger is just to illustrate this, I don’t think anyone thinks you should actually pay them (after all, you might encounter a more generous mugger tomorrow, or any number of more realistic opportunities to do astronomical amounts of good...)
Part of the issue is that I am coming at this problem from a different perspective than maybe you or Eliezer is. I believe that paying the mugger is basically worthless in the sense that doing almost any old good thing is better than paying the mugger. I would like to have a satisfying explanation of this. In contrast, Eliezer is interested in reconciling a view about complexity priors with a view about utility functions, and the mugger is an illustration of the conflict.
I do not have a proposed reconciliation of complexity priors and unbounded utility functions. Instead, the above comment is a recommended as an explanation of why paying the mugger is basically worthless in comparison with ordinary things you could do. So this hypothesis would say that if you set up your priors and your utility function in a reasonable way, the expected utility of downstream effects of ordinary good actions would greatly exceed the expected utility of paying the mugger.
Even if you decided that the expected utility framework somehow breaks down in cases like this, I think various related claims would still be plausible. E.g., rather than saying that doing ordinary good things has higher expected utility, it would be plausible that doing ordinary good things is “better relative to your uncertainty” than paying the mugger.
On a different note, another thing I find unsatisfying about the downstream effects reply is that it doesn’t seem to match up with why ordinary people think it is dumb to pay the mugger. The ultimate reason I think it is dumb to pay the mugger is strongly related to why ordinary people think it is dumb to pay the mugger, and I would like to be able to thoroughly understand the most plausible common-sense explanation of why paying the mugger is dumb. The proposed relationship between ordinary actions and their distant effects seems too far off from why common sense would say that paying the mugger is dumb. I guess this is ultimately pretty close to one of Nick Bostrom’s complaints about empirical stabilizing assumptions.
I think we are all in agreement with this (modulo the fact that all of the expected values end up being infinite and so we can’t compare in the normal way; if you e.g. proposed a cap of 3^^^^^^^3 on utilities, then you certainly wouldn’t pay the mugger).
It seems very likely to me that ordinary people are best modeled as having bounded utility functions, which would explain the puzzle.
So it seems like there are two issues:
You would never pay the mugger in any case, because other actions are better.
If you object to the fact that the only thing you care about is a very small probability of an incredibly good outcome, then that’s basically the definition of having a bounded utility function.
And then there is the third issue Eliezer is dealing with, where he wants to be able to have an unbounded utility function even if that doesn’t describe anyone’s preferences (since it seems like boundedness is an unfortunate restriction to randomly impose on your preferences for technical reasons), and formally it’s not clear how to do that. At the end of the post he seems to suggest giving up on that though.
Obviously to really put the idea of people having bounded utility functions to the test, you have to forget about it solving problems of small probabilities and incredibly good outcomes and focus on the most unintuitive consequences of it. For one, having a bounded utility function means caring arbitrarily little about differences between the goodness of different sufficiently good outcomes. And all the outcomes could be certain too. You could come up with all kinds of thought experiments involving purchasing huge numbers of years happy life or some other good for a few cents. You know all of this so I wonder why you don’t talk about it.
Also I believe that Eliezer thinks that an unbounded utility function describes at least his preferences. I remember he made a comment about caring about new happy years of life no matter how many he’d already been granted.
(I haven’t read most of the discussion in this thread or might just be missing something so this might be irrelevant.)
As far as I know the strongest version of this argument is Benja’s, here (which incidentally seems to deserve many more upvotes than it got).
Benja’s scenario isn’t a problem for normal people though, who are not reflectively consistent and whose preferences manifestly change over time.
Beyond that, it seems like people’s preferences regarding the lifespan dilemma are somewhat confusing and probably inconsistent, much like their preferences regarding the repugnant conclusion. But that seems mostly orthogonal to pascal’s mugging, and the basic point—having unbounded utility by definition means you are willing to accept negligible chances of sufficiently good outcomes against probability nearly 1 of any fixed bad outcome, so if you object to the latter you are just objecting to unbounded utility.
I agree I was being uncharitable towards Eliezer. But it is true that at the end of this post he was suggesting giving up on unbounded utility, and that everyone in this crowd seems to ultimately take that route.
Sorry, I didn’t mean to suggest otherwise. The “different perspective” part was supposed to be about the “in contrast” part.
I agree with yli that this has other unfortunate consequences. And, like Holden, I find it unfortunate to have to say that saving N lives with probability 1/N is worse than saving 1 life with probability 1. I also recognize that the things I would like to say about this collection of cases are inconsistent with each other. It’s a puzzle. I have written about this puzzle at reasonable length in my dissertation. I tend to think that bounded utility functions are the best consistent solution I know of, but that continuing to operate with inconsistent preferences (in a tasteful way) may be better in practice.