I’ll need some background here. Why aren’t bounded utilities the default assumption? You’d need some extraordinary arguments to convince me that anyone has an unbounded utility function. Yet this post and many others on LW seem to implicitly assume unbounded utility functions.
1) We don’t need an unbounded utility function to demonstrate Pascal’s Mugging. Plain old large numbers like 10^100 are enough.
2) It seems reasonable for utility to be linear in things we care about, e.g. human lives. This could run into a problem with non-uniqueness, i.e., if I run an identical computer program of you twice, maybe that shouldn’t count as two. But I think this is sufficiently murky as to not make bounded utility clearly correct.
We don’t need an unbounded utility function to demonstrate Pascal’s Mugging. Plain old large numbers like 10^100 are enough.
The scale is arbitrary. If your utility function is designed such that utility for common scenario are not very small compared to the maximum utility then you wouldn’t have Pascal’s Muggings.
It seems reasonable for utility to be linear in things we care about, e.g. human lives.
Does anybody really have linear preferences in anything? This seems at odds with empirical evidence.
Why aren’t bounded utilities the default assumption?
Because here the default utility is the one specified by the Von Neumann-Morgenstern theorem and there is no requirement (or indication) that it is bounded.
Humans, of course, don’t operate according to VNM axioms, but most of LW thinks it’s a bug to be fixed X-/
Does it? As far as I know, all it says is that the utility function exists. Maybe it’s bounded or maybe not—VNM does not say.
VNM main theorem proves that if you have a set of preferences consistent with some requirements, then an utility function exists such that maximizing its expectation satisfies your preferences.
If you are designing an agent ex novo, you can choose a bounded utility function. This restricts the set of allowed preferences, in a way that essentially prevents Pascal’s Mugging.
I don’t think it would because the bounds are arbitrary and if you make them wide enough, Pascal’s Mugging will still work perfectly well.
Yes, but if the expected utility for common scenarios is not very far from the bounds, then Pascal’s Mugging will not apply.
you can choose a bounded utility function. This restricts the set of allowed preferences
How does that work? VNM preferences are basically ordering or ranking. What kind of VNM preferences would be disallowed under a bounded utility function?
if the expected utility for common scenarios is not very far from the bounds, then Pascal’s Mugging will not apply
Are you saying that you can/should set the bounds narrowly? You lose your ability to correctly react to rare events, then—and black swans are VERY influential.
VNM preferences are basically ordering or ranking.
Only in the deterministic case. If you have uncertainty, this doesn’t apply anymore: utility is invariant to positive affine transforms, not to arbitrary monotone transforms.
What kind of VNM preferences would be disallowed under a bounded utility function?
Any risk-neutral (or risk-seeking) preference in any quantity.
If you have uncertainty, this doesn’t apply anymore
I am not sure I understand. Uncertainty in what? Plus, if you are going beyond the VNM Theorem, what is the utility function we’re talking about, anyway?
In the outcome of each action. If the world is deterministic, then all that matters is a preference ranking over outcomes. This is called ordinal utility.
If the outcomes for each action are sampled from some action-dependent probability distribution, then a simple ranking isn’t enough to express your preferences. VNM theory allows you to specify a cardinal utility function, which is invariant only up to positive affine transform.
In practice this is needed to model common human preferences like risk-aversion w.r.t. money.
If the outcomes for each action are sampled from some action-dependent probability distribution, then a simple ranking isn’t enough to express your preference.
Yes, you need risk tolerance / risk preference as well, but once we have that, aren’t we already outside of the VNM universe?
Consistent risk preferences can be encapsulated in the shape of the utility function—preferring a certain $40 to a half chance of $100 and half chance of nothing, for example, is accomplished by a broad class of utility functions. Preferences on probabilities—treating 95% as different than midway between 90% and 100%--cannot be expressed in VNM utility, but that seems like a feature, not a bug.
In principle, utility non-linear in money produces various amounts of risk aversion or risk seeking. However, this fundamental paper proves that observed levels of risk aversion cannot be thus explained. The results have been generalised here to a class of preference theories broader than expected utility.
However, this fundamental paper proves that observed levels of risk aversion cannot be thus explained.
This paper has come up before, and I still don’t think it proves anything of the sort. Yes, if you choose crazy inputs a sensible function will have crazy outputs—why did this get published?
In general, prospect theory is a better descriptive theory of human decision-making, but I think it makes for a terrible normative theory relative to utility theory. (This is why I specified consistent risk preferences—yes, you can’t express transaction or probabilistic framing effects in utility theory. As said in the grandparent, that seems like a feature, not a bug.)
Because here the default utility is the one specified by the Von Neumann-Morgenstern theorem and there is no requirement (or indication) that it is bounded.
Except, the VNM theorem in the form given applies to situations with finitely many possibilities. If there are infinitely many possibilities, then the generalized theorem does require bounded utility. This follows from precisely the Pascal’s mugging-type arguments like the ones being considered here.
Except, the VNM theorem in the form given applies to situations with finitely many possibilities.
In the page cited a proof outline is given for the finite case, but the theorem itself has no such restriction, whether “in the form given” or, well, the theorem itself.
If there are infinitely many possibilities, then the generalized theorem does require bounded utility.
What are you referring to as the generalised theorem? Something other than the one that VNM proved? That certainly does not require or assume bounded utility.
This follows from precisely the Pascal’s mugging-type arguments like the ones being considered here.
If you’re referring to the issue in the paper that entirelyuseless cited, Lumifer correctly pointed out that it is outside the setting of VNM (and someone downvoted him for it).
The paper does raise a real issue, though, for the setting it discusses. Bounding the utility is one of several possibilities that it briefly mentions to salvage the concept.
The paper is also useful in clarifying the real problem of Pascal’s Mugger. It is not that you will give all your money away to strangers promising 3^^^3 utility. It is that the calculation of utility in that setting is dominated by extremes of remote possibility of vast positive and negative utility, and nowhere converges.
Physicists ran into something of the sort in quantum mechanics, but I don’t know if the similarity is any more than superficial, or if the methods they worked out to deal with it have any analogue here.
What are you referring to as the generalised theorem?
Try this:
Theorem: Using the notation from here, except we will allow lotteries to have infinitely many outcomes as long as the probabilities sum to 1.
If an ordering satisfies the four axioms of completeness, transitivity, continuity, and independence, and the following additional axiom:
Axiom (5): Let L = Sum(i=0...infinity, p_i M_i) with Sum(i=0...infinity, p_i)=1 and N >= Sum(i=0...n, p_i M_i)/Sum(i=0...n, p_i) then N >= L. And similarly with the arrows reversed.
An agent satisfying axioms (1)-(5) has preferences given by a bounded utility function u such that, L>M iff Eu(L)>Eu(M).
Axiom (5): Let L = Sum(i=0...infinity, pi Mi) with Sum(i=0...infinity, pi)=1 and N >= Sum(i=0...n, pi Mi)/Sum(i=0...n, pi) then N >= L. And similarly with the arrows reversed.
That appears to be an axiom that probabilities go to zero enough faster than utilities that total utility converges (in a setting in which the sure outcomes are a countable set). It lacks something in precision of formulation (e.g. what is being quantified over, and in what order?) but it is fairly clear what it is doing. There’s nothing like it in VNM’s book or the Wiki article, though. Where does it come from?
Yes, in the same way that VNM’s axioms are just what is needed to get affine utilities, an axiom something like this will give you bounded utilities. Does the axiom have any intuitive appeal, separate from it providing that consequence? If not, the axiom does not provide a justification for bounded utilities, just an indirect way of getting them, and you might just as well add an axiom saying straight out that utilities are bounded.
None of which solves the problem that entirelyuseless cited. The above axiom forbids the Solomonoff prior (for which pi Mi grows with busy beaver fastness), but does not suggest any replacement universal prior.
That appears to be an axiom that probabilities go to zero enough faster than utilities that total utility converges (in a setting in which the sure outcomes are a countable set).
No, the axiom doesn’t put any constraints on the probability distribution. It merely constrains preferences, specifically it says that preferences for infinite lotteries should be the ‘limits’ of the preference for finite lotteries. One can think of it as a slightly stronger version of the following:
Axiom (5′): Let L = Sum(i=0...infinity, p_i M_i) with Sum(i=0...infinity, p_i)=1. Then if for all i N>=M_I then N>=L. And similarly with the arrows reversed. (In other words if N is preferred over every element of a lottery then N is preferred over the lottery.)
In fact, I’m pretty sure that axiom (5′) is strong enough, but I haven’t worked out all the details.
It lacks something in precision of formulation (e.g. what is being quantified over, and in what order?)
Sorry, there were some formatting problems, hopefully it’s better now.
(for which p_i M_i [formatting fixed] grows with busy beaver fastness)
The M_i’s are lotteries that the agent has preferences over, not utility values. Thus it doesn’t a priori make sense to talk about its growth rate.
I think I understand what the axiom is doing. I’m not sure it’s strong enough, though. There is no guarantee that there is any N that is >= M_i for all i (or for all large enough i, a weaker version which I think is what is needed), nor an N that is ⇐ them. But suppose there are such an upper Nu and a lower Nl, thus giving a continuous range between them of Np = p Nl + (1-p) Nu for all p in 0..1. There is no guarantee that the supremum of those p for which Np is a lower bound is equal to the infimum of those for which it is an upper bound. The axiom needs to stipulate that lower and upper bounds Nl and Nu exist, and that there is no gap in the behaviours of the family Np.
One also needs some axioms to the effect that a formal infinite sum Sum{i>=0: pi Mi} actually behaves like one, otherwise “Sum” is just a suggestively named but uninterpreted symbol. Such axioms might be invariance under permutation, equivalence to a finite weighted average when only finitely many pi are nonzero, and distribution of the mixture process to the components for infinite lotteries having the same sequence of component lotteries. I’m not sure that this is yet strong enough.
The task these axioms have to perform is to uniquely extend the preference relation from finite lotteries to infinite lotteries. It may be possible to do that, but having thought for a while and not come up with a suitable set of axioms, I looked for a counterexample.
Consider the situation in which there is exactly one sure-thing lottery M. The infinite lotteries, with the axioms I suggested in the second paragraph, can be identified with the probability distributions over the non-negative integers, and they are equivalent when they are permutations of each other. All of the distributions with finite support (call these the finite lotteries) are equivalent to M, and must be assigned the same utility, call it u. Take any distribution with infinite support, and assign it an arbitrary utility v. This determines the utility of all lotteries that are weighted averages of that one with M. But that won’t cover all lotteries yet. Take another one and give it an arbitrary utility w. This determines the utility of some more lotteries. And so on. I don’t think any inconsistency is going to arise. This allows for infinitely many different preference orderings, and hence infinitely many different utility functions.
The construction is somewhat analogous to constructing an additive function from reals to reals, i.e. one satisfying f(a+b) = f(a) + f(b). The only continuous additive functions are multiplication by a constant, but there are infinitely many non-continuous additive functions.
An alternative approach would be to first take any preference ordering consistent with the axioms, then use the VNM axioms to construct a utility function for that preference ordering, and then to impose an axiom about the behaviour of that utility function, because once we have utilities it’s easy to talk about limits. The most straightforward such axiom would be to stipulate that U( Sum{i>=0: pi Mi} ) = Sum{i>=0: pi U(Mi)}, where the sum on the right hand side is an ordinary infinite sum of real numbers. The axiom would require this to converge.
This axiom has the immediate consequence that utilities are bounded, for if they were not, then for any probability distribution {i>=0: pi} with infinite support, one could choose a sequence of lotteries whose utilities grew fast enough that Sum{i>=0: pi U(Mi)} would fail to converge.
Personally, I am not convinced that bounded utility is the way to go to avoid Pascal’s Mugging, because I see no principled way to choose the bound. The larger you make it, the more Muggings you are vulnerable to, but the smaller you make it, the more low-hanging fruit you will ignore: substantial chances of stupendous rewards.
In one of Eliezer’s talks, he makes a point about how bad an existential risk to humanity is. It must be measured not by the number of people who die in it when it happens, but the loss of a potentially enormous future of humanity spreading to the stars. That is the real difference between “only” 1 billion of us dying, and all 7 billion. If you are moved by this argument, you must see a substantial gap between the welfare of 7 billion people and that of however many 10^n you foresee if we avoid these risks. That already gives substantial headroom for Muggings.
I think I understand what the axiom is doing. I’m not sure it’s strong enough, though. There is no guarantee that there is any N that is >= M_i for all i (or for all large enough i, a weaker version which I think is what is needed), nor an N that is ⇐ them.
The M_i’s can themselves be lotteries. The idea is to group events into finite lotteries so that the M_i’s are >= N.
Personally, I am not convinced that bounded utility is the way to go to avoid Pascal’s Mugging, because I see no principled way to choose the bound.
There is no principled way to chose utility functions either, yet people seem to be fine with them.
My point is that if one takes the VNM theory seriously as justification for having a utility function, the same logic means it must be bounded.
There is no principled way to chose utility functions either, yet people seem to be fine with them.
The VNM axioms are the principled way. That’s not to say that it’s a way I agree with, but it is a principled way. The axioms are the principles, codifying an idea of what it means for a set of preferences to be rational. Preferences are assumed given, not chosen.
My point is that if one takes the VNM theory seriously as justification for having a utility function, the same logic means it must be bounded.
Boundedness does not follow from the VNM axioms. It follows from VNM plus an additional construction of infinite lotteries, plus additional axioms about infinite lotteries such as those we have been discussing. Basically, if utilities are unbounded, then there are St. Petersburg-style infinite lotteries with divergent utilities; if all infinite lotteries are required to have defined utilities, then utilities are bounded.
This is indeed a problem. Either utilities are bounded, or some infinite lotteries have no defined value. When probabilities are given by algorithmic probability, the situation is even worse: if utilities are unbounded then no expected utiilties are defined.
But the problem is not solved by saying, “utilities must be bounded then”. Perhaps utilities must be bounded. Perhaps Solomonoff induction is the wrong way to go. Perhaps infinite lotteries should be excluded. (Finitists would go for that one.) Perhaps some more fundamental change to the conceptual structure of rational expectations in the face of uncertainty is called for.
They show that you must have a utility function, not what it should be.
Boundedness does not follow from the VNM axioms. It follows from VNM plus an additional construction of infinite lotteries, plus additional axioms about infinite lotteries such as those we have been discussing.
Well the additional axiom is as intuitive as the VNM ones, and you need infinite lotteries if you are too model a world with infinite possibilities.
Perhaps Solomonoff induction is the wrong way to go.
This amounts to rejecting completeness. Suppose omega offered to create a universe based on a Solomonoff prior, you’d have to way to evaluate this proposal.
They show that you must have a utility function, not what it should be.
Given your preferences, they do show what your utility function should be (up to affine transformation).
Well the additional axiom is as intuitive as the VNM ones, and you need infinite lotteries if you are too model a world with infinite possibilities.
You need some, but not all of them.
This amounts to rejecting completeness.
By completeness I assume you mean assigning a finite utility to every lottery, including the infinite ones. Why not reject completeness? The St. Petersburg lottery is plainly one that cannot exist. I therefore see no need to assign it any utility.
Bounded utility does not solve Pascal’s Mugging, it merely offers an uneasy compromise between being mugged by remote promises of large payoffs and passing up unremote possibilities of large payoffs.
Suppose omega offered to create a universe based on a Solomonoff prior, you’d have to way to evaluate this proposal.
I don’t care. This is a question I see no need to have any answer to. But why invoke Omega? The Solomonoff prior is already put forward by some as a universal prior, and it is already known to have problems with unbounded utility. As far as I know this problem is still unsolved.
Actually, I would, but that’s digressing from the subject of infinite lotteries. As I have been pointing out, infinite lotteries are outside the scope of the VNM axioms and need additional axioms to be defined. It seems no more reasonable to me to require completeness of the preference ordering over St. Petersburg lotteries than to require that all sequences of real numbers converge.
Care to assign a probability to that statement.
“True.” At some point, probability always becomes subordinate to logic, which knows only 0 and 1. If you can come up with a system in which it’s probabilities all the way down, write it up for a mathematics journal.
If you’re going to cite this (which makes a valid point, but people usually repeat the password in place of understanding the idea), tell me what probability you assign to A conditional on A, to 1+1=2, and to an omnipotent God being able to make a weight so heavy he can’t lift it.
“True.” At some point, probability always becomes subordinate to logic, which knows only 0 and 1. If you can come up with a system in which it’s probabilities all the way down, write it up for a mathematics journal.
Ok, so care to present an a priori pure logic argument for why St. Petersburg lottery-like situations can’t exist.
Ok, so care to present an a priori pure logic argument for why St. Petersburg lottery-like situations can’t exist.
FInite approximations to the St. Petersburg lottery have unbounded values. The sequence does not converge to a limit.
In contrast, a sequence of individual gambles with expectations 1, 1⁄2, 1⁄4, etc. does have a limit, and it is reasonable to allow the idealised infinite sequence of them a place in the set of lotteries.
You might as well ask why the sum of an infinite number of ones doesn’t exist. There are ways of extending the real numbers with various sorts of infinite numbers, but they are extensions. The real numbers do not include them. The difficulty of devising an extension that allows for the convergence of all infinite sums is not an argument that the real numbers should be bounded.
Our main result implies that if you have an unbounded, perception determined, computable utility function, and you use a Solomonoff-like prior (Solomonoff, 1964), then you have no way to choose between policies using expected utility.
So, it’s within the AIXI context and you feed your utility function infinite (!) sequences of “perceptions”.
I’ll need some background here. Why aren’t bounded utilities the default assumption? You’d need some extraordinary arguments to convince me that anyone has an unbounded utility function. Yet this post and many others on LW seem to implicitly assume unbounded utility functions.
1) We don’t need an unbounded utility function to demonstrate Pascal’s Mugging. Plain old large numbers like 10^100 are enough.
2) It seems reasonable for utility to be linear in things we care about, e.g. human lives. This could run into a problem with non-uniqueness, i.e., if I run an identical computer program of you twice, maybe that shouldn’t count as two. But I think this is sufficiently murky as to not make bounded utility clearly correct.
The scale is arbitrary. If your utility function is designed such that utility for common scenario are not very small compared to the maximum utility then you wouldn’t have Pascal’s Muggings.
Does anybody really have linear preferences in anything? This seems at odds with empirical evidence.
Like V_V, I don’t find it “reasonable” for utility to be linear in things we care about.
I will write a discussion topic about the issue shortly.
EDIT: Link to the topic: http://lesswrong.com/r/discussion/lw/mv3/unbounded_linear_utility_functions/
Because here the default utility is the one specified by the Von Neumann-Morgenstern theorem and there is no requirement (or indication) that it is bounded.
Humans, of course, don’t operate according to VNM axioms, but most of LW thinks it’s a bug to be fixed X-/
But VNM theory allows for bounded utility functions, so if we are designing an agent why don’t design it with a bounded utility function?
It would systematically solve Pascal’s Mugging, and more formally, it would prevent the expectations to ever become undefined.
Does it? As far as I know, all it says is that the utility function exists. Maybe it’s bounded or maybe not—VNM does not say.
I don’t think it would because the bounds are arbitrary and if you make them wide enough, Pascal’s Mugging will still work perfectly well.
VNM main theorem proves that if you have a set of preferences consistent with some requirements, then an utility function exists such that maximizing its expectation satisfies your preferences.
If you are designing an agent ex novo, you can choose a bounded utility function. This restricts the set of allowed preferences, in a way that essentially prevents Pascal’s Mugging.
Yes, but if the expected utility for common scenarios is not very far from the bounds, then Pascal’s Mugging will not apply.
How does that work? VNM preferences are basically ordering or ranking. What kind of VNM preferences would be disallowed under a bounded utility function?
Are you saying that you can/should set the bounds narrowly? You lose your ability to correctly react to rare events, then—and black swans are VERY influential.
Only in the deterministic case. If you have uncertainty, this doesn’t apply anymore: utility is invariant to positive affine transforms, not to arbitrary monotone transforms.
Any risk-neutral (or risk-seeking) preference in any quantity.
I am not sure I understand. Uncertainty in what? Plus, if you are going beyond the VNM Theorem, what is the utility function we’re talking about, anyway?
In the outcome of each action. If the world is deterministic, then all that matters is a preference ranking over outcomes. This is called ordinal utility.
If the outcomes for each action are sampled from some action-dependent probability distribution, then a simple ranking isn’t enough to express your preferences. VNM theory allows you to specify a cardinal utility function, which is invariant only up to positive affine transform.
In practice this is needed to model common human preferences like risk-aversion w.r.t. money.
Yes, you need risk tolerance / risk preference as well, but once we have that, aren’t we already outside of the VNM universe?
No, risk tolerance / risk preference can be modeled with VNM theory.
Link?
Consistent risk preferences can be encapsulated in the shape of the utility function—preferring a certain $40 to a half chance of $100 and half chance of nothing, for example, is accomplished by a broad class of utility functions. Preferences on probabilities—treating 95% as different than midway between 90% and 100%--cannot be expressed in VNM utility, but that seems like a feature, not a bug.
In principle, utility non-linear in money produces various amounts of risk aversion or risk seeking. However, this fundamental paper proves that observed levels of risk aversion cannot be thus explained. The results have been generalised here to a class of preference theories broader than expected utility.
This paper has come up before, and I still don’t think it proves anything of the sort. Yes, if you choose crazy inputs a sensible function will have crazy outputs—why did this get published?
In general, prospect theory is a better descriptive theory of human decision-making, but I think it makes for a terrible normative theory relative to utility theory. (This is why I specified consistent risk preferences—yes, you can’t express transaction or probabilistic framing effects in utility theory. As said in the grandparent, that seems like a feature, not a bug.)
Except, the VNM theorem in the form given applies to situations with finitely many possibilities. If there are infinitely many possibilities, then the generalized theorem does require bounded utility. This follows from precisely the Pascal’s mugging-type arguments like the ones being considered here.
In the page cited a proof outline is given for the finite case, but the theorem itself has no such restriction, whether “in the form given” or, well, the theorem itself.
What are you referring to as the generalised theorem? Something other than the one that VNM proved? That certainly does not require or assume bounded utility.
If you’re referring to the issue in the paper that entirelyuseless cited, Lumifer correctly pointed out that it is outside the setting of VNM (and someone downvoted him for it).
The paper does raise a real issue, though, for the setting it discusses. Bounding the utility is one of several possibilities that it briefly mentions to salvage the concept.
The paper is also useful in clarifying the real problem of Pascal’s Mugger. It is not that you will give all your money away to strangers promising 3^^^3 utility. It is that the calculation of utility in that setting is dominated by extremes of remote possibility of vast positive and negative utility, and nowhere converges.
Physicists ran into something of the sort in quantum mechanics, but I don’t know if the similarity is any more than superficial, or if the methods they worked out to deal with it have any analogue here.
Try this:
Theorem: Using the notation from here, except we will allow lotteries to have infinitely many outcomes as long as the probabilities sum to 1.
If an ordering satisfies the four axioms of completeness, transitivity, continuity, and independence, and the following additional axiom:
Axiom (5): Let L = Sum(i=0...infinity, p_i M_i) with Sum(i=0...infinity, p_i)=1 and N >= Sum(i=0...n, p_i M_i)/Sum(i=0...n, p_i) then N >= L. And similarly with the arrows reversed.
An agent satisfying axioms (1)-(5) has preferences given by a bounded utility function u such that, L>M iff Eu(L)>Eu(M).
Edit: fixed formatting.
That appears to be an axiom that probabilities go to zero enough faster than utilities that total utility converges (in a setting in which the sure outcomes are a countable set). It lacks something in precision of formulation (e.g. what is being quantified over, and in what order?) but it is fairly clear what it is doing. There’s nothing like it in VNM’s book or the Wiki article, though. Where does it come from?
Yes, in the same way that VNM’s axioms are just what is needed to get affine utilities, an axiom something like this will give you bounded utilities. Does the axiom have any intuitive appeal, separate from it providing that consequence? If not, the axiom does not provide a justification for bounded utilities, just an indirect way of getting them, and you might just as well add an axiom saying straight out that utilities are bounded.
None of which solves the problem that entirelyuseless cited. The above axiom forbids the Solomonoff prior (for which pi Mi grows with busy beaver fastness), but does not suggest any replacement universal prior.
No, the axiom doesn’t put any constraints on the probability distribution. It merely constrains preferences, specifically it says that preferences for infinite lotteries should be the ‘limits’ of the preference for finite lotteries. One can think of it as a slightly stronger version of the following:
Axiom (5′): Let L = Sum(i=0...infinity, p_i M_i) with Sum(i=0...infinity, p_i)=1. Then if for all i N>=M_I then N>=L. And similarly with the arrows reversed. (In other words if N is preferred over every element of a lottery then N is preferred over the lottery.)
In fact, I’m pretty sure that axiom (5′) is strong enough, but I haven’t worked out all the details.
Sorry, there were some formatting problems, hopefully it’s better now.
The M_i’s are lotteries that the agent has preferences over, not utility values. Thus it doesn’t a priori make sense to talk about its growth rate.
I think I understand what the axiom is doing. I’m not sure it’s strong enough, though. There is no guarantee that there is any N that is >= M_i for all i (or for all large enough i, a weaker version which I think is what is needed), nor an N that is ⇐ them. But suppose there are such an upper Nu and a lower Nl, thus giving a continuous range between them of Np = p Nl + (1-p) Nu for all p in 0..1. There is no guarantee that the supremum of those p for which Np is a lower bound is equal to the infimum of those for which it is an upper bound. The axiom needs to stipulate that lower and upper bounds Nl and Nu exist, and that there is no gap in the behaviours of the family Np.
One also needs some axioms to the effect that a formal infinite sum Sum{i>=0: pi Mi} actually behaves like one, otherwise “Sum” is just a suggestively named but uninterpreted symbol. Such axioms might be invariance under permutation, equivalence to a finite weighted average when only finitely many pi are nonzero, and distribution of the mixture process to the components for infinite lotteries having the same sequence of component lotteries. I’m not sure that this is yet strong enough.
The task these axioms have to perform is to uniquely extend the preference relation from finite lotteries to infinite lotteries. It may be possible to do that, but having thought for a while and not come up with a suitable set of axioms, I looked for a counterexample.
Consider the situation in which there is exactly one sure-thing lottery M. The infinite lotteries, with the axioms I suggested in the second paragraph, can be identified with the probability distributions over the non-negative integers, and they are equivalent when they are permutations of each other. All of the distributions with finite support (call these the finite lotteries) are equivalent to M, and must be assigned the same utility, call it u. Take any distribution with infinite support, and assign it an arbitrary utility v. This determines the utility of all lotteries that are weighted averages of that one with M. But that won’t cover all lotteries yet. Take another one and give it an arbitrary utility w. This determines the utility of some more lotteries. And so on. I don’t think any inconsistency is going to arise. This allows for infinitely many different preference orderings, and hence infinitely many different utility functions.
The construction is somewhat analogous to constructing an additive function from reals to reals, i.e. one satisfying f(a+b) = f(a) + f(b). The only continuous additive functions are multiplication by a constant, but there are infinitely many non-continuous additive functions.
An alternative approach would be to first take any preference ordering consistent with the axioms, then use the VNM axioms to construct a utility function for that preference ordering, and then to impose an axiom about the behaviour of that utility function, because once we have utilities it’s easy to talk about limits. The most straightforward such axiom would be to stipulate that U( Sum{i>=0: pi Mi} ) = Sum{i>=0: pi U(Mi)}, where the sum on the right hand side is an ordinary infinite sum of real numbers. The axiom would require this to converge.
This axiom has the immediate consequence that utilities are bounded, for if they were not, then for any probability distribution {i>=0: pi} with infinite support, one could choose a sequence of lotteries whose utilities grew fast enough that Sum{i>=0: pi U(Mi)} would fail to converge.
Personally, I am not convinced that bounded utility is the way to go to avoid Pascal’s Mugging, because I see no principled way to choose the bound. The larger you make it, the more Muggings you are vulnerable to, but the smaller you make it, the more low-hanging fruit you will ignore: substantial chances of stupendous rewards.
In one of Eliezer’s talks, he makes a point about how bad an existential risk to humanity is. It must be measured not by the number of people who die in it when it happens, but the loss of a potentially enormous future of humanity spreading to the stars. That is the real difference between “only” 1 billion of us dying, and all 7 billion. If you are moved by this argument, you must see a substantial gap between the welfare of 7 billion people and that of however many 10^n you foresee if we avoid these risks. That already gives substantial headroom for Muggings.
The M_i’s can themselves be lotteries. The idea is to group events into finite lotteries so that the M_i’s are >= N.
There is no principled way to chose utility functions either, yet people seem to be fine with them.
My point is that if one takes the VNM theory seriously as justification for having a utility function, the same logic means it must be bounded.
The VNM axioms are the principled way. That’s not to say that it’s a way I agree with, but it is a principled way. The axioms are the principles, codifying an idea of what it means for a set of preferences to be rational. Preferences are assumed given, not chosen.
Boundedness does not follow from the VNM axioms. It follows from VNM plus an additional construction of infinite lotteries, plus additional axioms about infinite lotteries such as those we have been discussing. Basically, if utilities are unbounded, then there are St. Petersburg-style infinite lotteries with divergent utilities; if all infinite lotteries are required to have defined utilities, then utilities are bounded.
This is indeed a problem. Either utilities are bounded, or some infinite lotteries have no defined value. When probabilities are given by algorithmic probability, the situation is even worse: if utilities are unbounded then no expected utiilties are defined.
But the problem is not solved by saying, “utilities must be bounded then”. Perhaps utilities must be bounded. Perhaps Solomonoff induction is the wrong way to go. Perhaps infinite lotteries should be excluded. (Finitists would go for that one.) Perhaps some more fundamental change to the conceptual structure of rational expectations in the face of uncertainty is called for.
They show that you must have a utility function, not what it should be.
Well the additional axiom is as intuitive as the VNM ones, and you need infinite lotteries if you are too model a world with infinite possibilities.
This amounts to rejecting completeness. Suppose omega offered to create a universe based on a Solomonoff prior, you’d have to way to evaluate this proposal.
Given your preferences, they do show what your utility function should be (up to affine transformation).
You need some, but not all of them.
By completeness I assume you mean assigning a finite utility to every lottery, including the infinite ones. Why not reject completeness? The St. Petersburg lottery is plainly one that cannot exist. I therefore see no need to assign it any utility.
Bounded utility does not solve Pascal’s Mugging, it merely offers an uneasy compromise between being mugged by remote promises of large payoffs and passing up unremote possibilities of large payoffs.
I don’t care. This is a question I see no need to have any answer to. But why invoke Omega? The Solomonoff prior is already put forward by some as a universal prior, and it is already known to have problems with unbounded utility. As far as I know this problem is still unsolved.
Assuming your preferences satisfy the axioms.
No, by completeness I mean that for any two lotteries you prefer one over the other.
So why not reject it in the finite case as well?
Care to assign a probability to that statement.
Actually, I would, but that’s digressing from the subject of infinite lotteries. As I have been pointing out, infinite lotteries are outside the scope of the VNM axioms and need additional axioms to be defined. It seems no more reasonable to me to require completeness of the preference ordering over St. Petersburg lotteries than to require that all sequences of real numbers converge.
“True.” At some point, probability always becomes subordinate to logic, which knows only 0 and 1. If you can come up with a system in which it’s probabilities all the way down, write it up for a mathematics journal.
If you’re going to cite this (which makes a valid point, but people usually repeat the password in place of understanding the idea), tell me what probability you assign to A conditional on A, to 1+1=2, and to an omnipotent God being able to make a weight so heavy he can’t lift it.
Ok, so care to present an a priori pure logic argument for why St. Petersburg lottery-like situations can’t exist.
FInite approximations to the St. Petersburg lottery have unbounded values. The sequence does not converge to a limit.
In contrast, a sequence of individual gambles with expectations 1, 1⁄2, 1⁄4, etc. does have a limit, and it is reasonable to allow the idealised infinite sequence of them a place in the set of lotteries.
You might as well ask why the sum of an infinite number of ones doesn’t exist. There are ways of extending the real numbers with various sorts of infinite numbers, but they are extensions. The real numbers do not include them. The difficulty of devising an extension that allows for the convergence of all infinite sums is not an argument that the real numbers should be bounded.
They have unbounded expected values, that doesn’t mean the St. Petersburg lottery can’t exist, only that its expected value doesn’t.
(And with finitely many possibilities, the utility function cannot possibly be unbounded, because any finite set of reals has a maximum.)
I am not sure I understand. Link?
http://arxiv.org/pdf/0907.5598.pdf
So, it’s within the AIXI context and you feed your utility function infinite (!) sequences of “perceptions”.
We’re not in VNM land any more.