Why wait for the mugger to make his stupendous offer? Maybe he’s going to give you this stupendous blessing anyway—can you put a sufficiently low probability on that? Don’t you have to give all your money to the next person you meet? But wait! Maybe instead he intends to inflict unbounded negative utility if you do that—what must you do to be saved from that fate? Maybe the next rock you see is a superintelligent, superpowerful alien who, for its superunintelligible reasons requires you to—well, you get the idea.
The difference between this and the standard Mugger scenario is that by making his offer, the mugger promotes to attention the hypothesis that he presents. However, for the usual Bayesian reasons, this must at the same time promote many other unlikely hypotheses, such as the mugger being an evil tempter. I don’t see any reason to suppose that the mugger’s claim promotes any of these hypotheses sufficiently to distinguish the two scenarios. If you’re vulnerable to Pascal’s Mugger, you’ve already been mugged by your own decision theory.
If your decision theory has you walking through the world obsessed with tiny possibilities of vast utility fluctuations, like a placid-seeming vacuum state seething with colossal energies, then your decision theory is wrong. I propose the following constraint on utility-based rational decision theories:
The Anti-Mugging Axiom: For events E and current knowledge X, let P(E|X) = probability of E given X, U(E|X) = utility of E given X. For every state of knowledge X, P(E|X) U(E|X) is bounded over all events E.
The quantifiers here are deliberately chosen. For each X there must be an upper bound, but no bound is placed on the amount of probability-weighted utility that one might discover.
Well, it’s been two-and-a-quarter years since that post, but I’ll comment anyway.
Isn’t the anti-mugging axiom inadequate as stated? Basically, you’re saying the expected utility is bounded, but bounded by what? If the bound is, for example, equivalent to 20 happy years of life, you’re going to get mugged until you can barely keep from starving. If it’s less than 20 happy years of life, you probably won’t bother saving for retirement (assuming I’m interpreting this correctly).
Another way of looking at it, is that, let’s say the bound is b, then U(E|X) < b/P(E|X) ∀ X, ∀ E. So an event you’re sure will happen can have maximum utility b, but an event that you’re much less confident about can have vastly higher maximum utilities. This seems unintuitive (which is not as much of an issue as the one stated above).
Perhaps a stronger version is necessary. How about this: P(E|X) U(E|X) should tend to zero as U(E|X) tends to infinity. Or to put that with more mathematical clarity:
For any sequence of hypothetical events E_i, i=0, 1, …, if the sequence of utilities U(E_i|X) tends to infinity then the sequence of expectations P(E_i|X) U(E_i|X) must tend to zero.
Or perhaps an even stronger “uniform” version: For every e > 0 there exists a utility u such that for every event E with U(E|X) > u, its expected utility P(E|X) U(E|X) is less than e.
I called this an axiom, but it would be more accurate to call it a principle, something that any purported decision theory should satisfy as a theorem.
Hm, to be honest, I can’t quite wrap my head around the first version. Specifically, we’re choosing any sequence of events whatsoever, then if the utilities of the sequence tend to infinity (presumably equivalent to “increase without bound”, or maybe “increase monotonically without bound”?), then the expected utilities have to tend to zero? I feel like there’s not enough description of the early parts of the sequence. E.g. if it starts off as “going for a walk in nice weather, reading a mediocre book, kissing someone you like, inheriting a lot of money from a relative you don’t know or care about as you expected to do, accomplishing something really impressive...”, are we supposed to reduce probabilities on this part too? And if not, then do we start when we’re threatened with 3^^^3 disutilons, or only if it’s 3^^^^3 or more, or something?
I don’t think the second version works without setting further restrictions either, although I’m not entirely sure. E.g. choose u = (3^^^^3)^2/e, then clearly u is monotonically decreasing in e, so by the time we get to e = 3^^^^3, we get (approximately) that “an event with utility around 3^^^^3 can have utility at most 3^^^^3” with no further restrictions (since all previous e-u pairs have higher u’s, and therefore do not apply to this particular event), so that doesn’t actually help us any.
Anyway, it took me something like 20 minutes to decide on that, which mostly suggests that it’s been too long since I did actual math. I think the most reasonable and simple solution is to just have a bounded utility function (with the main question of interest being what sort of bound is best). There are definitely some alternative, more complicated, solutions, but we’d have to figure out in what (if any) ways they are actually superior.
Here’s another variation on the theme. Pascal’s Reformed Mugger comes to you and offers you, one time only, any amount of utility you ask for, but asks nothing in return.
If you believe him enough that u*P(you’ll get u by asking for it) is unbounded, how much do you ask for?
Do you also have to consider -u*P(you’ll get -u by asking for u)?
This is similar to the formulation I gave here, but I don’t think your version works. You could construct a series of different sets of knowledge X(n) that differ only in that they have different numbers n plugged in, and a bounding function B(n) such that
for all n P(E|X(n))U(E|X(n)) < B(n), but
lim[n->inf] P(E|X(n))U(E|X(n)) = inf
Basically, the mugger gets around your bound by crafting a state of knowledge X for you.
I’m pretty sure the formulation given in my linked comment also protects against Pascal’s Reformed Mugger.
Basically, the mugger gets around your bound by crafting a state of knowledge X for you.
This is giving too much power to the hypothetical mugger. If he can make me believe (I should have called X prior belief rather than prior knowledge) anything he chooses, then I don’t have anything. My entire state of mind is what it is only at his whim. Did you intend something less than this?
One could strengthen the axiom by requiring a bound on P(E|X) U(E|X) uniform in both E and X. However, if utiilty is unbounded, this implies that there is an amount so great that I can never believe it is attainable, even if it is. A decision theory that a priori rules out belief in something that could be true is also flawed.
There would have to be statements X(n) such that the maximum over E of
P(E|The mugger said X(n)) U(E|The mugger said X(n)) is unbounded in n. I don’t see why there should be, even if the maximum over E of P(E|X) U(E|X) is unbounded in X.
There would have to be statements X(n) such that the maximum over E of P(E|The mugger said X(n)) U(E|The mugger said X(n)) is unbounded in n.
Yes, and that is precisely what I said causes vulnerability to Pascal’s Mugging and should therefore be forbidden. Does your version of the anti-mugging axiom ensure that no such X exists, and can you prove it mathematically?
It does not ensure that no such X exists, but I think this scenario is outside the scope of your suggestion, which is expressed in terms of P(X) and U(X), rather than conditional probabilities and utilities.
What do you think of the other potential defect in a decision theory resulting from too strong an anti-mugging axiom: the inability to believe in the possibility of a sufficiently large amount of utility, regardless of any evidence?
Oh, so that’s where the confusion is coming from; the probabilities and utilities in my formulation are conditional, I just chose the notation poorly. Since X is a function of type number=>evidence-set, P(X(n)) means the probability of something (which I never assigned a variable name) given X(n), and U(X(n)) is the utility of that same thing given X. Giving that something a name, as in your notation, these would be P(E|X) and U(E|X).
Being unable to believe in sufficiently large amounts of utility regardless of any evidence would be very bad; we need to be careful not to phrase our anti-mugging defenses in ways that would do that. This is a problem with globally bounded utility functions, for example. I’m pretty sure that requiring all parameterized statements to produce expected utility that does not diverge to infinity as the parameter increases, does not cause any such problems.
Why wait for the mugger to make his stupendous offer? Maybe he’s going to give you this stupendous blessing anyway—can you put a sufficiently low probability on that? Don’t you have to give all your money to the next person you meet? But wait! Maybe instead he intends to inflict unbounded negative utility if you do that—what must you do to be saved from that fate? Maybe the next rock you see is a superintelligent, superpowerful alien who, for its superunintelligible reasons requires you to—well, you get the idea.
The difference between this and the standard Mugger scenario is that by making his offer, the mugger promotes to attention the hypothesis that he presents. However, for the usual Bayesian reasons, this must at the same time promote many other unlikely hypotheses, such as the mugger being an evil tempter. I don’t see any reason to suppose that the mugger’s claim promotes any of these hypotheses sufficiently to distinguish the two scenarios. If you’re vulnerable to Pascal’s Mugger, you’ve already been mugged by your own decision theory.
If your decision theory has you walking through the world obsessed with tiny possibilities of vast utility fluctuations, like a placid-seeming vacuum state seething with colossal energies, then your decision theory is wrong. I propose the following constraint on utility-based rational decision theories:
The Anti-Mugging Axiom: For events E and current knowledge X, let P(E|X) = probability of E given X, U(E|X) = utility of E given X. For every state of knowledge X, P(E|X) U(E|X) is bounded over all events E.
The quantifiers here are deliberately chosen. For each X there must be an upper bound, but no bound is placed on the amount of probability-weighted utility that one might discover.
Well, it’s been two-and-a-quarter years since that post, but I’ll comment anyway.
Isn’t the anti-mugging axiom inadequate as stated? Basically, you’re saying the expected utility is bounded, but bounded by what? If the bound is, for example, equivalent to 20 happy years of life, you’re going to get mugged until you can barely keep from starving. If it’s less than 20 happy years of life, you probably won’t bother saving for retirement (assuming I’m interpreting this correctly).
Another way of looking at it, is that, let’s say the bound is b, then U(E|X) < b/P(E|X) ∀ X, ∀ E. So an event you’re sure will happen can have maximum utility b, but an event that you’re much less confident about can have vastly higher maximum utilities. This seems unintuitive (which is not as much of an issue as the one stated above).
Perhaps a stronger version is necessary. How about this: P(E|X) U(E|X) should tend to zero as U(E|X) tends to infinity. Or to put that with more mathematical clarity:
For any sequence of hypothetical events E_i, i=0, 1, …, if the sequence of utilities U(E_i|X) tends to infinity then the sequence of expectations P(E_i|X) U(E_i|X) must tend to zero.
Or perhaps an even stronger “uniform” version: For every e > 0 there exists a utility u such that for every event E with U(E|X) > u, its expected utility P(E|X) U(E|X) is less than e.
I called this an axiom, but it would be more accurate to call it a principle, something that any purported decision theory should satisfy as a theorem.
Hm, to be honest, I can’t quite wrap my head around the first version. Specifically, we’re choosing any sequence of events whatsoever, then if the utilities of the sequence tend to infinity (presumably equivalent to “increase without bound”, or maybe “increase monotonically without bound”?), then the expected utilities have to tend to zero? I feel like there’s not enough description of the early parts of the sequence. E.g. if it starts off as “going for a walk in nice weather, reading a mediocre book, kissing someone you like, inheriting a lot of money from a relative you don’t know or care about as you expected to do, accomplishing something really impressive...”, are we supposed to reduce probabilities on this part too? And if not, then do we start when we’re threatened with 3^^^3 disutilons, or only if it’s 3^^^^3 or more, or something?
I don’t think the second version works without setting further restrictions either, although I’m not entirely sure. E.g. choose u = (3^^^^3)^2/e, then clearly u is monotonically decreasing in e, so by the time we get to e = 3^^^^3, we get (approximately) that “an event with utility around 3^^^^3 can have utility at most 3^^^^3” with no further restrictions (since all previous e-u pairs have higher u’s, and therefore do not apply to this particular event), so that doesn’t actually help us any.
Anyway, it took me something like 20 minutes to decide on that, which mostly suggests that it’s been too long since I did actual math. I think the most reasonable and simple solution is to just have a bounded utility function (with the main question of interest being what sort of bound is best). There are definitely some alternative, more complicated, solutions, but we’d have to figure out in what (if any) ways they are actually superior.
Here’s another variation on the theme. Pascal’s Reformed Mugger comes to you and offers you, one time only, any amount of utility you ask for, but asks nothing in return.
If you believe him enough that u*P(you’ll get u by asking for it) is unbounded, how much do you ask for?
Do you also have to consider -u*P(you’ll get -u by asking for u)?
This is similar to the formulation I gave here, but I don’t think your version works. You could construct a series of different sets of knowledge X(n) that differ only in that they have different numbers n plugged in, and a bounding function B(n) such that
Basically, the mugger gets around your bound by crafting a state of knowledge X for you.
I’m pretty sure the formulation given in my linked comment also protects against Pascal’s Reformed Mugger.
This is giving too much power to the hypothetical mugger. If he can make me believe (I should have called X prior belief rather than prior knowledge) anything he chooses, then I don’t have anything. My entire state of mind is what it is only at his whim. Did you intend something less than this?
One could strengthen the axiom by requiring a bound on P(E|X) U(E|X) uniform in both E and X. However, if utiilty is unbounded, this implies that there is an amount so great that I can never believe it is attainable, even if it is. A decision theory that a priori rules out belief in something that could be true is also flawed.
He doesn’t get to make you believe anything he chooses; making you believe statements of the form “The mugger said X(n)” is entirely sufficient.
There would have to be statements X(n) such that the maximum over E of P(E|The mugger said X(n)) U(E|The mugger said X(n)) is unbounded in n. I don’t see why there should be, even if the maximum over E of P(E|X) U(E|X) is unbounded in X.
Yes, and that is precisely what I said causes vulnerability to Pascal’s Mugging and should therefore be forbidden. Does your version of the anti-mugging axiom ensure that no such X exists, and can you prove it mathematically?
It does not ensure that no such X exists, but I think this scenario is outside the scope of your suggestion, which is expressed in terms of P(X) and U(X), rather than conditional probabilities and utilities.
What do you think of the other potential defect in a decision theory resulting from too strong an anti-mugging axiom: the inability to believe in the possibility of a sufficiently large amount of utility, regardless of any evidence?
Oh, so that’s where the confusion is coming from; the probabilities and utilities in my formulation are conditional, I just chose the notation poorly. Since X is a function of type number=>evidence-set, P(X(n)) means the probability of something (which I never assigned a variable name) given X(n), and U(X(n)) is the utility of that same thing given X. Giving that something a name, as in your notation, these would be P(E|X) and U(E|X).
Being unable to believe in sufficiently large amounts of utility regardless of any evidence would be very bad; we need to be careful not to phrase our anti-mugging defenses in ways that would do that. This is a problem with globally bounded utility functions, for example. I’m pretty sure that requiring all parameterized statements to produce expected utility that does not diverge to infinity as the parameter increases, does not cause any such problems.