You said that you would be okay with losing $5 to a mugger who raised your posterior by a factor of 10^(500), because they would have to do a lot of work to do so. I responded by pointing out that they wouldn’t have to do much work after all. If this doesn’t change the decision problem (which I agree with) then I don’t see how your original reasoning that it’s okay to get mugged because the mugger would have to work hard to mug you makes any sense.
What determines how much I am willing to pay is not how hard the mugger works per se, but how credible the threat is compared to its severity. (I thought this went without saying, and that you would be able to automatically generalize from “the mugger working hard” to “the mugger’s credibility increasing by whatever means”.) Going from p = 10^(-1000) to p = 10^(-500) may not sound like a “huge” increase in credibility, but it is. Or at least, if you insist that it isn’t, then you also have to concede that going from p = 10^(-500) to p = 1⁄2 isn’t that big of a credibility increase either, because it’s the same number of bits. In fact, measured in bits, going from p = 10^(-1000) to p = 10^(-500) is one-third of the way to p = 1-10^(-500) !
Now I presume you understand this arithmetic, so I agree that this is a distraction. In the same way, I think the simple mathematical arguments that you have been presenting are also a distraction. The real issue is that you apparently don’t believe that there exist outcomes with utilities in the range of 10^(750). Well, I am undecided on that question, because at this point I don’t know what “my” values look like in the limit of superintelligent extrapolation on galactic scales. (I like to think I’m pretty good at introspection, but I’m not that good!) But there’s no way I’m going to be convinced that my utility function has necessarily to be bounded without some serious argument going significantly beyond the fact that the consequences of an unbounded utility function seem counterintuitive to another human whose style of thought has already been demonstrated to be different from my own.
If you’ve got serious, novel arguments to offer for why a human-extracted utility function must be bounded, I’m quite willing to consider them, of course. But as of now I don’t have much evidence that you do have such arguments, because as far as I can tell, all you’ve said so far is “I can’t imagine anything with such high utility!”
P.S. Given that we’ve apparently had protracted disagreements on two issues so far, I just wanted you to know that I’m not trying to troll you or anything (in fact, I hadn’t realized that you were the same person who had made the Amanda Knox post). I will try to keep in mind in the future that our thinking styles are different and that appeals to intuition will probably just result in frustration.
What determines how much I am willing to pay is not how hard the mugger works per se, but how credible the threat is compared to its severity. (I thought this went without saying, and that you would be able to automatically generalize from “the mugger working hard” to “the mugger’s credibility increasing by whatever means”.) Going from p = 10^(-1000) to p = 10^(-500) may not sound like a “huge” increase in credibility, but it is. Or at least, if you insist that it isn’t, then you also have to concede that going from p = 10^(-500) to p = 1⁄2 isn’t that big of a credibility increase either, because it’s the same number of bits. In fact, measured in bits, going from p = 10^(-1000) to p = 10^(-500) is one-third of the way to p = 1-10^(-500) !
Now I presume you understand this arithmetic, so I agree that this is a distraction. In the same way, I think the simple mathematical arguments that you have been presenting are also a distraction. The real issue is that you apparently don’t believe that there exist outcomes with utilities in the range of 10^(750). Well, I am undecided on that question, because at this point I don’t know what “my” values look like in the limit of superintelligent extrapolation on galactic scales. (I like to think I’m pretty good at introspection, but I’m not that good!) But there’s no way I’m going to be convinced that my utility function has necessarily to be bounded without some serious argument going significantly beyond the fact that the consequences of an unbounded utility function seem counterintuitive to another human whose style of thought has already been demonstrated to be different from my own.
If you’ve got serious, novel arguments to offer for why a human-extracted utility function must be bounded, I’m quite willing to consider them, of course. But as of now I don’t have much evidence that you do have such arguments, because as far as I can tell, all you’ve said so far is “I can’t imagine anything with such high utility!”
Fair enough.
P.S. Given that we’ve apparently had protracted disagreements on two issues so far, I just wanted you to know that I’m not trying to troll you or anything (in fact, I hadn’t realized that you were the same person who had made the Amanda Knox post). I will try to keep in mind in the future that our thinking styles are different and that appeals to intuition will probably just result in frustration.