And if that new information includes an astronomically large number, then your new probability estimate ought to be infinitesimally small in a way that cancels out that astronomically large number.
Er, can you prove that? It doesn’t seem at all obvious to me that magic power improbability and magic power utility are directly proportional. Any given computation’s optimization power isn’t bounded in one to one correspondence by its Kolmogorov complexity as far as I can see, because that computation can still reach into other computations and flip sign bits that cause extremely widespread effects without being very complex itself. If you think there’s even a small chance that you’re in a computation susceptible to intervention by probable but powerful computations like that, then it’s not obvious that the improbability and potential utility cancel out.
Sorry for not responding earlier; I had to think about this a bit. Whether the presence of astronomically large numbers can make you vulnerable to Pascal’s Mugging seems to be a property of the interaction between the method you use to assign probabilities from evidence, and your utility function. Call the probability-assignment method P(X), which takes a statement X and returns a probability; and the utility function U(X), which assigns a utility to something (such as the decision to pay the mugger) based on the assumption that X is true.
P and U are vulnerable to Pascal’s Mugging if and only if you can construct sets of evidence X(n), which differ only by a single number n, such that for any utility value u, there exists n such that P(X(n))U(X(n)) > u.
Now, I really don’t know of any reason apart from Pascal’s Mugging why utility function-predictor pairs should have this property. But being vulnerable to Pascal’s Mugging is such a serious flaw, I’m tempted to say that it’s just a necessary requirement for mental stability, so any utility function and predictor which don’t guarantee this when they’re combined should be considered incompatible.
But being vulnerable to Pascal’s Mugging is such a serious flaw, I’m tempted to say that it’s just a necessary requirement for mental stability, so any utility function and predictor which don’t guarantee this when they’re combined should be considered incompatible.
Is the wording of this correct? Did you mean to say that vulnerability to Pascal’s mugging is a necessary requirement for mental stability or the opposite?
I’m interpreting your stance as “the probability that your hypothesis matches the evidence is bounded by the utility it would give you if your hypothesis matched the evidence.” Reductio ad absurdum: I am conscious. Tautologically true. Being conscious is to me worth a ton of utility. I should therefore disbelieve a tautology.
Hmm, apparently that wasn’t as clearly expressed as I thought. Let’s try that again. I said that a predictor P and utility function U are vulnerable to Pascal’s mugging if
exists function X of type number => evidence-set
such that X(a) differs from X(b) only in that one number appearing literally, and
forall u exists n such that P(X(n))U(X(n)) > u
The last line is the delta-epsilon definition for limits diverging to infinity. It could be equivalently written as
lim[n->inf] P(X(n))U(X(n)) = inf
If that limit diverges to infinity, then you could scale the probability down arbitrarily far and the mugger will just give you a bigger n. But if it doesn’t diverge that way, then there’s a maximum amount of expected utility the mugger can offer you just by increasing n, and the only way to get around it would be to offer more evidence that wasn’t in X(n).
(Note that while the limit can’t diverge to infinity, it is not required to converge. For example, the Pebblesorter utility function, U(n pebbles) = if(isprime(n)) 1 else 0, does not converge when combined with the null predictor P(X)=0.5.)
(The reductio you gave in the other reply does not apply, because the high-utility statement you gave is not parameterized, so it can’t diverge.)
Er, can you prove that? It doesn’t seem at all obvious to me that magic power improbability and magic power utility are directly proportional. Any given computation’s optimization power isn’t bounded in one to one correspondence by its Kolmogorov complexity as far as I can see, because that computation can still reach into other computations and flip sign bits that cause extremely widespread effects without being very complex itself. If you think there’s even a small chance that you’re in a computation susceptible to intervention by probable but powerful computations like that, then it’s not obvious that the improbability and potential utility cancel out.
Goddammit Less Wrong the above is a brilliant counterargument and no one realizes it. I hate all of you.
Sorry for not responding earlier; I had to think about this a bit. Whether the presence of astronomically large numbers can make you vulnerable to Pascal’s Mugging seems to be a property of the interaction between the method you use to assign probabilities from evidence, and your utility function. Call the probability-assignment method P(X), which takes a statement X and returns a probability; and the utility function U(X), which assigns a utility to something (such as the decision to pay the mugger) based on the assumption that X is true.
P and U are vulnerable to Pascal’s Mugging if and only if you can construct sets of evidence X(n), which differ only by a single number n, such that for any utility value u, there exists n such that P(X(n))U(X(n)) > u.
Now, I really don’t know of any reason apart from Pascal’s Mugging why utility function-predictor pairs should have this property. But being vulnerable to Pascal’s Mugging is such a serious flaw, I’m tempted to say that it’s just a necessary requirement for mental stability, so any utility function and predictor which don’t guarantee this when they’re combined should be considered incompatible.
Is the wording of this correct? Did you mean to say that vulnerability to Pascal’s mugging is a necessary requirement for mental stability or the opposite?
No, I meant to say that immunity to Pascal’s mugging is required.
I’m interpreting your stance as “the probability that your hypothesis matches the evidence is bounded by the utility it would give you if your hypothesis matched the evidence.” Reductio ad absurdum: I am conscious. Tautologically true. Being conscious is to me worth a ton of utility. I should therefore disbelieve a tautology.
u is the integer returned by U for an input X? Just wanted to make sure; I’m crafting my response.
Edit: actually, I have no idea what determines u here, ’cuz if u is the int returned by U then your inequality is tautological. No?
Hmm, apparently that wasn’t as clearly expressed as I thought. Let’s try that again. I said that a predictor P and utility function U are vulnerable to Pascal’s mugging if
The last line is the delta-epsilon definition for limits diverging to infinity. It could be equivalently written as
If that limit diverges to infinity, then you could scale the probability down arbitrarily far and the mugger will just give you a bigger n. But if it doesn’t diverge that way, then there’s a maximum amount of expected utility the mugger can offer you just by increasing n, and the only way to get around it would be to offer more evidence that wasn’t in X(n).
(Note that while the limit can’t diverge to infinity, it is not required to converge. For example, the Pebblesorter utility function, U(n pebbles) = if(isprime(n)) 1 else 0, does not converge when combined with the null predictor P(X)=0.5.)
(The reductio you gave in the other reply does not apply, because the high-utility statement you gave is not parameterized, so it can’t diverge.)