I agree with your conclusions and really like the post. Nevertheless I would like to offer a defense of rejecting Pascal’s Mugging even with an unbounded utility function, although I am not all that confident that my defense is actually correct.
Warning: slight rambling and half-formed thoughts below. Continue at own risk.
If I wish to consider the probability of the event [a stranger blackmails me with the threat to harm 3↑↑↑3 people] we should have some grasp of the likeliness of the claim [there exists a person/being that can willfully harm 3↑↑↑3 people]. There are two reasons I can think of why this claim is so problematic that perhaps we should assign it on the order of 1/3↑↑↑3 probability.
Firstly while the Knuth up-arrow notation is cutely compact, I think it is helpful to consider the set of claims [there exists a being that can willfully harm n people] for each value of n. Each claim implies all those before it, so the probabilities of this sequence should be decreasing. From this point of view I find it not at all strange that the probability of this claim can make explicit reference to the number n and behave as something like 1/n. The point I’m trying to make is that while 3↑↑↑3 is only 5 symbols, the claim to be able to harm that many people is so much more extraordinary than a a lot of similar claims that we might be justified in assigning it really low probability. Compare it to an artifical lottery where we force our computer to draw a number between 0 and 10^10^27 (deliberately chosen larger than the ‘practical limit’ referred to in the post). I think we can justly assign the claim [The number 3 will be the winning lottery ticket] a probability of 1/10^10^27. Something similar is going on here: there are so many hypothesis about being able to harm n people that in order to sum the probabilities to 1 we are forced to assign on the order of 1/n probability to each of them.
Secondly (and I hope this reinforces the slight rambling above) consider how you might be convinced that this stranger can harm 3↑↑↑3 people, as opposed to only has the ability to harm 3↑↑↑3 − 1 people. I think the ‘tearing open the sky’ magic trick wouldn’t do it—this will increase our confidence in the stranger being very powerful by extreme amounts (or, more realistically, convince us that we’ve gone completely insane), but I see no reason why we would be forced to assign significant probability to this stranger being able to harm 3↑↑↑3 people, instead of ‘just’ 10^100 or 10^10^26 people or something. Or in more Bayesian terms—which evidence E is more likely if this stranger can harm 3↑↑↑3 people than if it can harm, say, only 10^100 people? Which E satisfies P(E|stranger can harm 3↑↑↑3 people) > P(E|stranger can harm 10^100 people but not 3↑↑↑3 people)? Any suggestions along the lines of ‘present 3↑↑↑3 people and punch them all in the nose’ justifies having a prior of 1/3↑↑↑3 for this event, since showing that many people really is evidence with that likelihood ratio. But, hypothetically, if all such evidence E are of this form, are our actions then not consistent with the infinitesimal prior, since we require this particular likelihood ratio before we consider the hypothesis likely?
I hope I haven’t rambled too much, but I think that the Knuth up-arrow notation is hiding the complexity of the claim in Pascal’s Mugging, and that the evidence required to convince me that a being really has the power to do as is claimed has a likelyhood ratio close to 3↑↑↑3:1.
It is not true that if someone has the ability to harm n people, then they also necessarily have the ability to harm exactly m of those people for any m<n, so it isn’t clear that P(there is someone who has the ability to harm n people) monotonically decreases as n increases. Unless you meant at least n people, in which case that’s true, but still irrelevent since it doesn’t even establish that this probability approaches a limit of 0, much less that it does so at any particular rate.
Compare it to an artifical lottery where we force our computer to draw a number between 0 and 10^10^27 (deliberately chosen larger than the ‘practical limit’ referred to in the post). I think we can justly assign the claim [The number 3 will be the winning lottery ticket] a probability of 1/10^10^27.
The probability that a number chosen randomly from the uniform distribution on integers from 0 to 10^10^27 is 3 is indeed 1/10^10^27, but I wouldn’t count that as an empirical hypothesis. Given any particular mechanism for producing integers that you are very confident implements a uniform distribution on the integers from 0 to 10^10^27, the probability you should assign to it producing 3 is still much higher than 1/10^10^27.
Or in more Bayesian terms—which evidence E is more likely if this stranger can harm 3↑↑↑3 people than if it can harm, say, only 10^100 people?
The stranger says so, and has established some credibility by providing very strong evidence for other a priori very implausible claims that they have made.
Any suggestions along the lines of ‘present 3↑↑↑3 people and punch them all in the nose’ justifies having a prior of 1/3↑↑↑3 for this event, since showing that many people really is evidence with that likelihood ratio.
That’s not evidence that is physically possible to present to a human, and I don’t see why you say its likelihood ratio is around 1:3↑↑↑3.
I think I will try writing my reply as a full post, this discussion is getting longer than is easy to fit as a set of replies. You are right that my above reply has some serious flaws.
I agree with your conclusions and really like the post. Nevertheless I would like to offer a defense of rejecting Pascal’s Mugging even with an unbounded utility function, although I am not all that confident that my defense is actually correct.
Warning: slight rambling and half-formed thoughts below. Continue at own risk.
If I wish to consider the probability of the event [a stranger blackmails me with the threat to harm 3↑↑↑3 people] we should have some grasp of the likeliness of the claim [there exists a person/being that can willfully harm 3↑↑↑3 people]. There are two reasons I can think of why this claim is so problematic that perhaps we should assign it on the order of 1/3↑↑↑3 probability.
Firstly while the Knuth up-arrow notation is cutely compact, I think it is helpful to consider the set of claims [there exists a being that can willfully harm n people] for each value of n. Each claim implies all those before it, so the probabilities of this sequence should be decreasing. From this point of view I find it not at all strange that the probability of this claim can make explicit reference to the number n and behave as something like 1/n. The point I’m trying to make is that while 3↑↑↑3 is only 5 symbols, the claim to be able to harm that many people is so much more extraordinary than a a lot of similar claims that we might be justified in assigning it really low probability. Compare it to an artifical lottery where we force our computer to draw a number between 0 and 10^10^27 (deliberately chosen larger than the ‘practical limit’ referred to in the post). I think we can justly assign the claim [The number 3 will be the winning lottery ticket] a probability of 1/10^10^27. Something similar is going on here: there are so many hypothesis about being able to harm n people that in order to sum the probabilities to 1 we are forced to assign on the order of 1/n probability to each of them.
Secondly (and I hope this reinforces the slight rambling above) consider how you might be convinced that this stranger can harm 3↑↑↑3 people, as opposed to only has the ability to harm 3↑↑↑3 − 1 people. I think the ‘tearing open the sky’ magic trick wouldn’t do it—this will increase our confidence in the stranger being very powerful by extreme amounts (or, more realistically, convince us that we’ve gone completely insane), but I see no reason why we would be forced to assign significant probability to this stranger being able to harm 3↑↑↑3 people, instead of ‘just’ 10^100 or 10^10^26 people or something. Or in more Bayesian terms—which evidence E is more likely if this stranger can harm 3↑↑↑3 people than if it can harm, say, only 10^100 people? Which E satisfies P(E|stranger can harm 3↑↑↑3 people) > P(E|stranger can harm 10^100 people but not 3↑↑↑3 people)? Any suggestions along the lines of ‘present 3↑↑↑3 people and punch them all in the nose’ justifies having a prior of 1/3↑↑↑3 for this event, since showing that many people really is evidence with that likelihood ratio. But, hypothetically, if all such evidence E are of this form, are our actions then not consistent with the infinitesimal prior, since we require this particular likelihood ratio before we consider the hypothesis likely?
I hope I haven’t rambled too much, but I think that the Knuth up-arrow notation is hiding the complexity of the claim in Pascal’s Mugging, and that the evidence required to convince me that a being really has the power to do as is claimed has a likelyhood ratio close to 3↑↑↑3:1.
It is not true that if someone has the ability to harm n people, then they also necessarily have the ability to harm exactly m of those people for any m<n, so it isn’t clear that P(there is someone who has the ability to harm n people) monotonically decreases as n increases. Unless you meant at least n people, in which case that’s true, but still irrelevent since it doesn’t even establish that this probability approaches a limit of 0, much less that it does so at any particular rate.
The probability that a number chosen randomly from the uniform distribution on integers from 0 to 10^10^27 is 3 is indeed 1/10^10^27, but I wouldn’t count that as an empirical hypothesis. Given any particular mechanism for producing integers that you are very confident implements a uniform distribution on the integers from 0 to 10^10^27, the probability you should assign to it producing 3 is still much higher than 1/10^10^27.
The stranger says so, and has established some credibility by providing very strong evidence for other a priori very implausible claims that they have made.
That’s not evidence that is physically possible to present to a human, and I don’t see why you say its likelihood ratio is around 1:3↑↑↑3.
I think I will try writing my reply as a full post, this discussion is getting longer than is easy to fit as a set of replies. You are right that my above reply has some serious flaws.