Random thoughts here, not highly confident in their correctness.
Why is the leverage penalty seen as something that needs to be added, isn’t it just the obviously correct way to do probability.
Suppose I want to calculate the probability that a race of aliens will descend from the skies and randomly declare me Overlord of Earth some time in the next year. To do this, I naturally go to Delphi to talk to the Oracle of Perfect Priors, and she tells me that the chance of aliens descending from the skies and declaring an Overlord of Earth in the next year is 0.0000007%.
If I then declare this to be my probability of become Overlord of Earth in an alien-backed coup, this is obviously wrong. Clearly I should multiply it by the probability that the aliens pick me, given that the aliens are doing this. There are about 7-billion people on earth, and updating on the existence of Overlord Declaring aliens doesn’t have much effect on that estimate, so my probability of being picked is about 1 in 7 billion, meaning my probability of being overlorded is about 0.0000000000000001%. Taking the former estimate rather than the latter is simply wrong.
Pascal’s mugging is a similar situation, only this time when we update on the mugger telling the truth, we radically change our estimate of the number of people who were ‘in the lottery’, all the way up to 3^^^^3. We then multiply 1/3^^^^3 by the probability that we live in a universe where Pascal’s muggings occur (which should be very small but not super-exponentially small). This gives you the leverage penalty straight away, no need to think about Tegmark multiverses. We were simply mistaken to not include it in the first place.
only this time when we update on the mugger telling the truth, we radically change our estimate of the number of people who were ‘in the lottery’, all the way up to 3^^^^3. We then multiply 1/3^^^^3 by the probability that we live in a universe where Pascal’s muggings occur
How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?
That’s part of why I dislike Robin Hanson’s original solution. That the tempting/blackmailing offer involves 3^^^^3 other people, and that you are also a person should be merely incidental to one particular illustration of the problem of Pascal’s Mugging—and as such it can’t be part of a solution to the core problem.
To replace this with something like “causal nodes”, as Eliezer mentions, might perhaps solve the problem. But I wish that we started talking about Clippy and his paperclips instead, so that the original illustration of the problem which involves incidental symmetries doesn’t mislead us into a “solution” overreliant on symmetries.
How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?
Clippy has some sort of prior over the number of paperclips that could possibly exist. Let this number be P. Conditioned on each value of P, Clippy evaluates the utility of the offer and the probability that it comes true.
In particular, for P < 3^^^^3, the conditional probability that the offer of 3^^^^3 paperclips is legit is 0. If some large number of paperclips exists, e.g. P = 2*3^^^^3, the offer might actually be viable with non-negligible probability, while its utility would be given by 3^^^^3/P. Note that this is always at most 1.
However, unless Clippy lives in a very strange universe, it thinks that P >= 3^^^^3 is very unlikely. So the expected utility will be bounded by Pr[P >= 3^^^^3] and will end up being very small.
How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?
First thought, I’m not at all sure that it does. Pascal’s mugging may still be a problem. This doesn’t seem to contradict what I said about the leverage penalty being the only correct approach, rather than a ‘fix’ of some kind, in the first case. Worryingly, if you are correct it may also not be a ‘fix’ in the sense of not actually fixing anything.
I notice I’m currently confused about whether the ‘causal nodes’ patch is justified by the same argument. I will think about it and hopefully find an answer.
Random thoughts here, not highly confident in their correctness.
Why is the leverage penalty seen as something that needs to be added, isn’t it just the obviously correct way to do probability.
Suppose I want to calculate the probability that a race of aliens will descend from the skies and randomly declare me Overlord of Earth some time in the next year. To do this, I naturally go to Delphi to talk to the Oracle of Perfect Priors, and she tells me that the chance of aliens descending from the skies and declaring an Overlord of Earth in the next year is 0.0000007%.
If I then declare this to be my probability of become Overlord of Earth in an alien-backed coup, this is obviously wrong. Clearly I should multiply it by the probability that the aliens pick me, given that the aliens are doing this. There are about 7-billion people on earth, and updating on the existence of Overlord Declaring aliens doesn’t have much effect on that estimate, so my probability of being picked is about 1 in 7 billion, meaning my probability of being overlorded is about 0.0000000000000001%. Taking the former estimate rather than the latter is simply wrong.
Pascal’s mugging is a similar situation, only this time when we update on the mugger telling the truth, we radically change our estimate of the number of people who were ‘in the lottery’, all the way up to 3^^^^3. We then multiply 1/3^^^^3 by the probability that we live in a universe where Pascal’s muggings occur (which should be very small but not super-exponentially small). This gives you the leverage penalty straight away, no need to think about Tegmark multiverses. We were simply mistaken to not include it in the first place.
How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?
That’s part of why I dislike Robin Hanson’s original solution. That the tempting/blackmailing offer involves 3^^^^3 other people, and that you are also a person should be merely incidental to one particular illustration of the problem of Pascal’s Mugging—and as such it can’t be part of a solution to the core problem.
To replace this with something like “causal nodes”, as Eliezer mentions, might perhaps solve the problem. But I wish that we started talking about Clippy and his paperclips instead, so that the original illustration of the problem which involves incidental symmetries doesn’t mislead us into a “solution” overreliant on symmetries.
Clippy has some sort of prior over the number of paperclips that could possibly exist. Let this number be P. Conditioned on each value of P, Clippy evaluates the utility of the offer and the probability that it comes true.
In particular, for P < 3^^^^3, the conditional probability that the offer of 3^^^^3 paperclips is legit is 0. If some large number of paperclips exists, e.g. P = 2*3^^^^3, the offer might actually be viable with non-negligible probability, while its utility would be given by 3^^^^3/P. Note that this is always at most 1.
However, unless Clippy lives in a very strange universe, it thinks that P >= 3^^^^3 is very unlikely. So the expected utility will be bounded by Pr[P >= 3^^^^3] and will end up being very small.
First thought, I’m not at all sure that it does. Pascal’s mugging may still be a problem. This doesn’t seem to contradict what I said about the leverage penalty being the only correct approach, rather than a ‘fix’ of some kind, in the first case. Worryingly, if you are correct it may also not be a ‘fix’ in the sense of not actually fixing anything.
I notice I’m currently confused about whether the ‘causal nodes’ patch is justified by the same argument. I will think about it and hopefully find an answer.
This sounds a little bit like it might depend on the choice of SSA vs. SIA.