Better impossibility result for unbounded utilities

[paulfchristiano]

This post slightly improves on the impossibility results in my last post and A paradox for tiny probabilities and enormous values. This is very similar to those arguments but with minor technical differences. I may refer people here if we’re discussing details of the argument against unbounded utilities, but for more interesting discussion you should check out those other writeups.

Summarizing the key differences from my last post:

• I replace the dominance principle $\forall i:A_i>B_i\Rightarrow \sum p_i{A}_i>\sum p_i{B}_i$ with $\forall i:A_i\ge B_i\Rightarrow \sum p_i{A}_i\ge \sum p_i{B}_i$.

• Assuming unbounded positive and negative utilities, I remove the assumption of Intermediate Mixtures.

Weakening dominance

We’ll represent preferences by a relation $\le$ over probability distributions over some implicit space of outcomes $\Omega$ (and we’ll identify outcomes with the constant probability distribution). Define $A<B$ to mean ($A\le B$ and not $B\le A$).

We’ll show that it’s impossible for $\le$ to satisfy four properties: unbounded utilities, intermediate mixtures, very weak dominance, and transitivity.

The properties

Unbounded utilities: there is an infinite sequence of outcomes $X_1,X_2,X_4,X_8,\dots$ each “more than twice as good”^[1] as the last. More formally, there exists an outcome $X_0$ such that:

• $X_0<X_1$

• $X_{2^k}<\frac{1}{2}{X}_{2^{k+1}}+\frac{1}{2}{X}_0$ for every $k$

That is: $X_1$ is not as good as a $\frac{1}{2}$ chance of $X_2$, which is not as good as a $\frac{1}{4}$ chance of $X_4$, which is not as good as a $\frac{1}{8}$ chance of $X_8$…

Intermediate mixtures: If $A<B$ and $0<p<1$,^[2] then $A<pA+(1-p)B<B$.

That is: a chance of a good outcome is strictly worse than a sure thing.^[3]

Weak lottery-lottery dominance: Let $A_0,A_1,A_2,\dots$, and $B_0,B_1,B_2,\dots$ be sequences of lotteries, and $p_0,p_1,p_2,\dots$ be a sequence of probabilities summing to 1. If $A_i\ge B_i$ for all $i$, then $\sum p_i{A}_i\ge \sum p_i{B}_i$.

Transitivity: if $A\ge B$ and $B\ge C$, then $A\ge C$.^[4]

Inconsistency proof

Consider the lottery $X_{\infty }:=\frac{1}{3}{X}_0+\frac{2}{9}{X}_1+\frac{4}{27}{X}_2+\frac{8}{81}{X}_4+\dots$, where each outcome $X_{2^k}$ is $\frac{2}{3}$ as likely as the one before. Intuitively this sum is “more infinite” than the usual St Petersburg lottery and this will be important for our proof.

By weak dominance we can replace $X_1$ by $X_0$ without making the lottery any better. Thus:

• $X_{\infty }\ge \frac{5}{9}{X}_0+\frac{4}{27}{X}_2+\frac{8}{81}{X}_4+\frac{16}{243}{X}_8+\dots$

We can rewrite this new lottery as a mixture:

• $X_{\infty }\ge \frac{1}{9}{X}_0+\frac{8}{27}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_2)+\frac{16}{81}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_4)+\frac{32}{243}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_8)+\dots$

We can apply bounded utilities to each parenthesized expression. Combining with weak dominance, we obtain:

• $X_{\infty }\ge \frac{1}{9}{X}_0+\frac{8}{27}{X}_1+\frac{16}{81}{X}_2+\frac{32}{243}{X}_4+\dots$

Write $X_{\infty }^2$ for the right hand side of this inequality.

• $X_{\infty }^2:=\frac{1}{9}{X}_0+\frac{8}{27}{X}_1+\frac{16}{81}{X}_2+\frac{32}{243}{X}_4+\dots$

Now note that $X_{\infty }=\frac{1}{4}{X}_0+\frac{3}{4}{X}_{\infty }^2$, despite the fact that $X_{\infty }\ge X_{\infty }^2$. If we knew $X_{\infty }^2>X_0$, then intermediate mixtures would imply that $X_{\infty }^2>X_{\infty }$, contradicting $X_{\infty }\ge X_{\infty }^2$.

So all that remains is to show that $X_{\infty }^2>X_0$. This will involve a bit of annoying arithmetic but it should feel pretty obvious given that all the outcomes in $X_{\infty }^2$ seems much better than $X_0$.

Combining $\frac{1}{2}{X}_0+\frac{1}{2}{X}_2\ge X_1$ with weak dominance and $X_1>X_0$ we get:

• $X_{\infty }^2\ge \frac{13}{27}{X}_0+\frac{1}{27}{X}_1+\frac{7}{81}{X}_2+\frac{32}{243}{X}_4+\frac{64}{729}{X}_8+\dots$

Then we write the right hand side as a mixture and apply unbounded utilities and weak dominance:

• $X_{\infty }^2\ge \frac{1}{27}{X}_1+\frac{14}{81}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_2)+\frac{64}{243}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_4)+\frac{128}{729}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_8)+\dots$

• $X_{\infty }^2\ge \frac{1}{27}{X}_1+\frac{14}{81}{X}_1+\frac{64}{243}{X}_1+\frac{128}{729}{X}_1+\dots$

• $X_{\infty }^2\ge X_1>X_0$

Leveraging negative utilities

To rule out unbounded utilities, we’ve made two substantive consistency assumptions: intermediate mixtures and weak lottery-lottery dominance. The assumption of intermediate mixtures is necessary: we could satisfy the other properties by simply being indifferent between all lotteries with infinite expectations.

But if we can have unboundedly good or unboundedly bad outcomes, then we can obtain a contradiction even without infinite mixtures. That is, we will show that there is no relation $\le$ satisfying transitivity, symmetric unbounded utilities, and weak outcome-lottery dominance.

I think that almost anyone who accepts unbounded utilities (in the informal sense) should accept symmetric unbounded utilities, so I think they probably need to reject weak outcome-lottery dominance (or stop defining preferences over arbitrary probability distributions). I find this pretty damning, but maybe others are more comfortable with it.

The properties

Transitivity. If $A\le B$ and $B\le C$, then $A\le C$.

Symmetric unbounded utilities: There is a pair of outcomes $X_0<X_1$ (i.e. it’s not the case that all pairs of outcomes are either incomparable or equal). Moreover:

• For any^[5] pair of outcomes $X^{+}>X$, there is outcome $X^{-}$ such that $\frac{2}{3}{X}^{+}+\frac{1}{3}{X}^{-}\le X$.

• For any pair of outcomes $X^{-}<X$ there is an outcome $X^{+}$ such that $\frac{2}{3}{X}^{-}+\frac{1}{3}{X}^{+}\ge X$.

In words: no matter how much better $X^{+}$ is than $X$, there’s always an $X^{-}$ that’s 2x “further away” from $X$ on the other side. By that we mean that a $\frac{1}{3}$ risk of moving from $X$ to $X^{-}$ can offset a $\frac{2}{3}$ chance of moving from $X$ to $X^{+}$. And no matter how bad an outcome $X^{-}$ is, there’s always an $X^{+}$ that’s 2x “further away” from $X$ on the other side.

Weak outcome-lottery dominance: Let $X$ be an outcome, let $A_0,A_1,A_2,\dots$ be a sequence of lotteries, and let $p_0,p_1,p_2,\dots$ be a sequence of probabilities summing to 1. If $A_i\ge X$ for all $i$, then $\sum p_i{A}_i\ge X$. Similarly, if $A_i\le X$ for all $i$, then $\sum p_i{A}_i\le X$.

Inconsistency proof

Define a sequence of outcomes $X_0,X_1,X_{-2},X_7,X_{-14},X_{31}\dots$ as follows:

• Pick $X_0<X_1$ arbitrarily.

• Take $X_{-2}$ to be the “reflection” of $X_1$ across $X_0$ as defined in symmetric unbounded utilities.

• Take $X_7$ to be the reflection of $X_{-2}$ across $X_1$.

• Take $X_{-14}$ to be the reflection of $X_7$ across $X_0$.

• Take $X_{31}$ to be the reflection of $X_{-14}$ across $X_1$.

• And so on.

Now define the lottery $X_{\pm \infty }=\frac{1}{2}{X}_1+\frac{1}{4}{X}_{-2}+\frac{1}{8}{X}_7+\frac{1}{16}{X}_{-14}+\frac{1}{32}{X}_{31}+\dots$

We can write $X_{\pm \infty }$ as the mixture:

• $\frac{3}{4}(\frac{2}{3}{X}_1+\frac{1}{3}{X}_{-2})+\frac{3}{16}(\frac{2}{3}{X}_7+\frac{1}{3}{X}_{-14})+\frac{3}{64}(\frac{2}{3}{X}_{31}+\frac{1}{3}{X}_{-62})\dots$

By Unbounded Utilities each of these terms is $\le X_0$. So by weak dominance, $X_{\pm \infty }\le X_0.$

But we can also write $X_{\pm \infty }$ as the mixture:

• $\frac{1}{2}{X}_1+\frac{3}{8}(\frac{2}{3}{X}_{-2}+\frac{1}{3}{X}_7)+\frac{3}{32}(\frac{2}{3}{X}_{-14}+\frac{1}{3}{X}_{31})+\dots$

By Unbounded Utilities each of these terms is $\ge X_1$. So by weak dominance, $X_{\pm \infty }\ge X_1$.

Now we have $X_0\ge X_{\pm \infty }\ge X_1$. By transitivity, $X_0\ge X_1$, contradicting $X_1>X_0$.

1. ^

   Of course the same argument would work if we replaced “good” with “bad.”

2. ^

   Note that we only actually need to apply this principle for $p=\frac{3}{4}$ and $p=\frac{8}{9}$ so a reader squeamish about very small probabilities need not be concerned. By making the argument with different numbers we could probably just fix $p=\frac{1}{2}$.

3. ^

   Despite appearing innocuous, this might be more “controversial” than very weak dominance. Many theories say that if $B$ is infinitely good, then it doesn’t matter whether I achieve $B$ with 100% probability or 50% probability (or 1% probability or 0.000001% probability...) I find this sufficiently unappealing to reject such theories out of hand, but each reader must pick their own poison.

4. ^

   I’m pretty sure this isn’t necessary for the proof, since we only use it as a convenience for short manipulations rather than to establish very long chains. That said, it makes the proof easier and it’s a pretty mild assumption.

5. ^

   We don’t really need this universal quantifier—it would be enough to define a single chain of escalating and alternative outcomes. But the quantitative properties we need out of the chain are somewhat arbitrary, so it seemed clearer to state an axiom capturing why unbounded utilities allow us to construct a particular kind of chain, rather than to directly posit a chain. That said, it results in a stronger assumption—for example it may be that there is an outcome $X_{\infty }$ which can offset any negative outcome, while still having a chain of increasingly large finite outcomes.

[paulfchristiano]

Here’s a more intuitive way to view the first proof (under very slightly different assumptions).

Suppose that Omega offers two lotteries: a St. Petersburg lottery $X_{\infty }$ and a “half chance of a St. Petersburg lottery” $\frac{1}{2}{X}_0+\frac{1}{2}{X}_{\infty }$. Suppose I draw an outcome from $X_{\infty }$, I see what it is, and then I’m given the option to switch to the other (currently uncertain) lottery.

No matter what finite value I see, it’s very easy to argue that I’m going to want to switch (and I’d want to switch even if I truncated the second St. Petersburg lottery after some finite number of steps). But then by a reasonable dominance or “sure thing” principle I might as well just switch before I even look at the outcome.

But that implies $\frac{1}{2}{X}_0+\frac{1}{2}{X}_{\infty }\ge X_{\infty }$, i.e. I’m just as happy with a half chance of a St. Petersburg lottery as a sure thing of a St. Petersburg lottery. And similarly I’m just as happy with an $\epsilon$ chance for any $\epsilon >0$. That violates Intermediate Mixtures (and is generally just a kind of bizarre preference to have).

[Slider]

If you truncate the second lottery so that the outcome that you got from the first one doesn’t appear then you would not want to switch.

I am a little confused as I read this setup multiple times as “We draw a an outcome ‘jackpot’ from St. Petersburg. Then you get to choose between ‘jackpot’ and ‘heads goat, tails jackpot’”. If goat were just an arbitrary condolence price then it might happen that jackpot<goat and it might happen that jackpot>goat. But goat is not an unconnected outcome but the shittiest thing that St Petersburg can spit out. So jackpot>=goat and I am fine staying with jackpot. And I think the setup intends that the lotteries are drawn “separately”, that you can get a diffrent thing out. But in that case it escapes me how rolling one die would make me update in any direction about disconnected dice even if they have the same makeup. That somebody wins the lottery doesn’t make me update the value of lottery tickets upwards.

To my mind “half St. Petersburg” has a transfinite expectation value which is smaller than St. Petersburg. The main gist how the argument goes forward is imply that transfinite values breaking things sufficiently mean they need to have equal value. If you can’t say > and can’t say < one option is that the things are equal but another option is that they are incoparable or can’t be compared by those methods one wishes to use.

[MichaelStJules]

Oh, nice! It seems more irrational to me to violate this “sure thing” principle than the axioms in your post, or at least, this comment makes it clear that you can get Dutch booked and money pumped if you do so. You have a Dutch book, since the strategy forces you to commit to switching to a lottery that’s stochastically dominated by a lottery available at the start that you previously held (assuming $X_0$ has identically 0 payoff). There’s also a money pump here, since Omega can offer you a new St. Petersburg lottery after you see the outcome of your previous lottery and charge you an arbitrarily large finite amount to switch.

Still, this kind of behaviour seems hard to exploit in practice, because someone needs to be able to offer you a finite unbounded lottery with infinite expected value (or something similar, if we aren’t using expected values).

[Toby_Ord]

This is an interesting theorem which helps illuminate the relationship between unbounded utilities and St Petersburg gambles. I particularly appreciate that you don’t make an explicit assumption that the values of gambles must be representable by real numbers which is very common, but unhelpful in a setting like this. However, I do worry a bit about the argument structure.

The St Petersburg gamble is a famously paradox-riddled case. That is, it is a very difficult case where it isn’t clear what to say, and many theories seem to produce outlandish results. When this happens, it isn’t so impressive to say that we can rule out an opposing theory because in that paradox-riddled situation it would lead to strange results. It strikes me as similar to saying that a rival theory leads to strange result in variable population-size cases so we can reject it (when actually, all theories do), or that it leads to strange results in infinite population cases (when again, all theories do).

Even if one had a proof that an alternative theory doesn’t lead to strange conclusions in the St Petersburg gamble, I don’t think this would count all that much in its favour. As it seems plausible to me that various rules of decision theory that were developed in the cleaner cases of finite possibility spaces (or well-behaved infinite spaces) need to be tweaked to account for more pathological possibility spaces. For a simple example, I’m sympathetic to the sure thing principle, but it directly implies that the St Petersburg Gamble is better than itself, because an unresolved gamble is better than a resolved one, no matter how the latter was resolved. My guess is that this means the sure thing principle needs to have its scope limited to exclude gambles whose value is higher than that of any of their resolutions.

[paulfchristiano]

I think that weak outcome-lottery dominance is inconsistent with transitivity + unbounded utilities in both directions (or unbounded utilities in one direction + the sure thing principle), rather than merely producing strange results. Though we could summarize “violates weak outcome-lottery dominance” as a strange result.

Violating weak outcome-lottery dominance means that a mix of gambles, each strictly better than a particular outcome X, can fail to be at least as good as X. If you give up on this property, or on transitivity, then even if you are assigning numbers you call “utilities” to actions I don’t think it’s reasonable to call them utilities in the decision-theoretic sense, and I’m comfortable saying that your procedure should no longer be described as “expected utility maximization.”

So I’d conclude that there simply don’t exist any preferences represented by unbounded utility functions (over the space of all lotteries), and that there is no patch to the notion of utility maximization that fixes this problem without giving up on some defining feature of EU maximization.

There may nevertheless be theories that are well-described as maximizing an unbounded utility function in some more limited situations. And there may well be preferences over a domain other than lotteries which are described intuitively by an unbounded utility function. (Though note that if you are only considering lotteries over a finite space then your utility function is necessarily bounded.) And although it seems somewhat less likely it could also be that in retrospect I will feel I was wrong about the defining features of EU maximization, and mixing together positive lotteries to get a negative lottery is actually consistent with its spirit.

I think it’s also worth observing that although St Petersburg cases are famously paradox-riddled, these cases seem overwhelmingly important on a conventional utilitarian view even before we consider any exotic hypotheses. Indeed, I personally became unhappy with unbounded utilities not because of impossibility results but because I tried to answer questions like “How valuable is it to accelerate technological progress?” or “How bad is it if unaligned AI takes over the world?” and immediately found that EU maximization with anything like “utility linear in population size” seemed to be unworkable in practice. I could find no sort of common-sensical regularization that let me get coherent answers out of these theories, and I’m not sure what it would look like in practice to try to use them to guide our actions.

[MichaelStJules]

I think the more important takeaway is that the (countable) sure thing principle and transitivity together rule out preferences allowing St. Petersburg-like lotteries, and so “unbounded” preferences.

I recommend https://​​onlinelibrary.wiley.com/​​doi/​​abs/​​10.1111/​​phpr.12704

It discusses more ways preferences allowing St. Petersburg-like lotteries seem irrational, like choosing dominated strategies, dynamic inconsistency and paying to avoid information. Furthermore, they argue that the arguments supporting the finite Sure Thing Principle are generally also arguments in favour of the Countable Sure Thing Principle, because they don’t depend on the number of possibilities in a lottery being finite. So, if you reject the Countable Sure Thing Principle, you should probably reject the finite one, too, and if you accept St. Petersburg-like lotteries, you need to in principle accept behaviour that seems irrational.

They also have a general vNM-like representation theorem, dropping the Archimedean/​continuity axiom, and replacing the Independence axiom with Countable Independence, and with transitivity and completeness, they get utility functions with values in lexicographically ordered ordinal sequences of bounded real utilities. (They say the sequences can have any ordinal to order them, but that seems wrong to me, since I’d think infinite length lexicographically ordered sequences get you St. Petersburg-like lotteries and violate Limitedness, but maybe I’m misunderstanding. EDIT: I think they meant you can have a an infinite sequence of dominated components, not an infinite sequence of dominating components, so you check the most important component first, and then the second, and continue for possibly infinitely many. Well-orderedness ensures there’s always a next one to check.)

[paulfchristiano]

Thanks for the reference, seems better than my post and I hadn’t seen it. (I think it’s the version of the argument I allude to in this comment.)

Note that if you have unboundedly positive and unboundedly negative outcomes, then you must also violate the weaker version of the countable sure thing principle with weak inequality instead of strict inequality. Violating the weak version of the sure thing principle seems much worse to me. And I think most proponents of unbounded preferences would advocate for them to point in both directions, so they run into this stronger problem.

That said, countability and strength of the sure thing principle are orthogonal: countable weak sure thing + finite strong sure thing --> countable strong sure thing. Negative utilities are only relevant as a response to someone who is OK saying “a 1% chance of an infinitely good outcome is just as good as a sure thing,” since that view is inconsistent if the 1% chance of an infinitely good outcome could be balanced out by a 2% chance of an infinitely bad outcome.

[Richard_Kennaway]

There’s a step in one of your proofs that I don’t quite follow. It seems to depend on some sort of continuity axiom, perhaps implicit in the notion of infinite lotteries. It’s where you go from

$X_{\infty }^2\ge \frac{1}{27}{X}_1+\frac{14}{81}{X}_1+\frac{64}{243}{X}_1+\frac{128}{729}{X}_1+...$

to

$X_{\infty }^2\ge X_1$

I can see that the sum of the infinite sequence of coefficients is 1, but as elsewhere in the post you’ve avoided ever computing infinite sums, as that sort of computation is not a thing in this setup, I’m not seeing the justification here.

ETA: I think I have dissolved my confusion. Expressions of the form $p_0{X}_0+p_1{X}_1+p_2{X}_2+\dots$, finite or infinite, denote probability distributions, and therefore e.g.$\frac{1}{2}X+\frac{1}{2}X$ denotes the same distribution as $X$. The infinite sequence that I cited is by definition the same thing as $X_1$.

[cousin_it]

Can we have unbounded utilities, and lotteries with infinite support, but probabilities always go down so fast that the sum (absolutely) converges, no matter what evidence we’ve seen?

[Vanessa Kosoy]

Yes, for example you can penalize the (initially Solomonoff-ish) prior probability of every hypothesis by a factor of $e^{-\beta (U_{max}-U_{min})}$ where $\beta >0$ is some constant, $U_{max}$ is the maximal expected utility of this hypothesis over all policies, and $U_{min}$ is the minimal (and you’d have to discard hypotheses for which one of those is already divergent, except maybe in cases where the difference is renormalizable somehow). This kind of thing was referred to as “leverage penalty” in a previous discussion. Personally I’m quite skeptical it’s useful, but maaaybe?

[LGS]

As several people mentioned in the comments on the other post, it feels like the most natural way to get around this is to only have beliefs which are finite-support probability distributions. If you do that, the paradoxes go away.

Of course, if we do that, then the set of our beliefs is no longer “complete” in the topological sense; that is, if we pick a distance metric between distributions (such as total variation distance), then for the set of finite-support distributions, Cauchy sequences do not necessarily converge.

This suggests that perhaps we can represent our beliefs as Cauchy sequences of finite-support distributions. The paradox can then be rephrased in this setting: it says that if you unbounded utilities, then a Cauchy sequence of beliefs might have a utility that does not converge (e.g. it might swing wildly between positive and negative, and the swings can even get worse and worse as we go further down the sequence, despite the sequence being Cauchy and hence “intuitively convergent”).

[paulfchristiano]

I mentioned this briefly in a footnote on the other post. The summary is that it’s not exactly clear to me what it means to have “unbounded utility functions” if you think there are only finitely many conceivable outcomes. Isn’t there then some best outcome, out of the ${10}^{30}$ that you think deserve non-zero probability?

Perhaps there could be infinitely many possible decisions, but that each decision involves only finitely many possible outcomes? But that seems implausible to me. For example, consider my parents making a decision about how to raise me—if there are infinitely many decisions I might face, then it seems like there are infinitely many possible outcomes from their decision. To me this seems worse than abstract worries about continuity.

And if there are infinitely many possible outcomes of a decision, what does it mean to force my beliefs to have finite support? If I just consider a single set of finitely-supported beliefs, what exactly am I doing? If I take limits, then as you point out we can end up back at the same paradox.

I guess the out here would be to represent outcomes as sequences of finitely supported probability distributions, effectively adding additional structure (that is presumably related to how that distribution came about). That means that I don’t need to be indifferent between two sequences with the same limit, I can care about that extra data.

This is the kind of thing I have in mind by abandoning probability theory and representing my uncertainty with some richer structure. I don’t find “sequence of finitely-supported probability distributions” particularly compelling but it seems like something you could try (and if you did it that way maybe you wouldn’t have to give up on probability theory, though as I suggested I suspect that’s where this road will end).

I guess the two questions, for that and any other proposal, would be: (i) where does this extra structure come from? what about my epistemic state determines how it gets represented as a sequence? (ii) are there any sensible preferences over the new enlarged space?

(I will probably make some posts in the future with more concrete examples of how totally messed up the “intuitive” unbounded utility functions are, which will hopefully make those concerns sharper.)

[LGS]

The way I was envisioning it, there would be infinitely possible outcomes but you could only have a belief about finitely many of them at one time.

I don’t think this is too outrageous—for example, if there were uncountably many possible outcomes then we all agree that (no matter the setup) there would be unmeasurable sets that you could not have a belief over.

The main motivation here is just that this is a mathematically nice way to set it up. For example, if the set of all possible outcomes is A, then conv(A) (the convex hull of A) will be the set of all finite-support probability distributions over A—it comes up naturally.

[More formal version: identify the set A as a subset of the vector space $R^A$ of functions from A to $R$, where each element x of A is identified with the characteristic function that returns 1 on input x and 0 otherwise. Then the convex hull of A can be defined as the intersection of all convex supersets of A (all of which are subsets of the vector space $R^A$). It is then a relatively straight-forward theorem that this convex hull of A happens to be exactly the set of all functions $A\to R$ such that (1) they return 0 on all but finitely many elements, (2) they have non-negative range, and (3) the sum of their non-zero outputs is exactly 1; in other words, the convex hull of A is exactly the set of finite-support probability distributions over the set A. My point here is merely that finite-support probability distributions came up naturally, even in the context of an infinite outcome space A, and even though the definition of convex hull did not explicitly mention finite supports in any way.]

Upon reflection, I agree that sequences of such finite-support distributions are a kind of an ugly hack. In particular, it’s not clear how to mix together two such sequences (i.e. how to take a convex combination of them, something we may want to do with our beliefs).

We can just stick to finite-support distributions themselves, without allowing sequences of them. (Perhaps a motivation could be that our finite brains can only think about finitely many plausible outputs at a time, or something like that). In that case, I think the main drawback is only that we cannot model St. Petersburg paradox. However, given your counterexamples, perhaps this is a feature rather than a bug...

[paulfchristiano]

I guess I’m confused about how to represent my current beliefs with a finitely-supported probability distribution. It looks to me like there are infinitely many ways the universe could be (in the sense that e.g. I could start listing them and never stop, or that there are functions $f:universes\to universes$ for which $f\left(U\right)$ is bigger than $U$ while still being plausible).

I don’t expect to enumerate all these infinitely many universes, but practically how am I supposed to think about my preferences if it feels like there are clearly infinitely many possible states of affairs?

[LGS]

Your comment gave me pause, and certainly makes me lean away from finite-support probability distributions somewhat.

However, if the problem is that you can actively generate more and more plausible universes without stop, then it does seem at some level like your belief structure is a sequence of finite-support probability distributions, doesn’t it? As you mentally generate more and more plausible universes, your belief gets updates to a distribution with larger and larger support. The main problem is just that “sequence of distributions” is a much uglier mathematical object than a single distribution.

Another thought: if you can actively mentally generate more and more possible universes, and if, in addition, the universes you generate have such large utilities that they become “more and more important” to consider (i.e. even after multiplying by their diminishing probabilities, the absolute value of probability*utility is increasing), then you are screwed. This was shown nicely by your examples. So in some sense, we have to restrict to situations where the possible universes you mentally generate are diminishing in importance (i.e. even if their utility is increasing, their probability is diminishing fast enough to make the sequence absolutely convergent).

[Slider]

Does this approach mean that questions like “How far I prefer my neighbours house to be from mine?” are still answereable?

[LGS]

If you believe that spacetime is discrete at the Planck scale, then there are only finitely many options for how far your neighbor’s house can be from yours. I tend to think that finite-support probability distributions are sufficient for this task… even if spacetime is continuous, we can get a good-enough approximation by assuming it is discrete at the Planck scale.

(Is there some context I’m missing here? I don’t know if I’m supposed to recognize your example.)

[Slider]

I am trying to ask about the limits of the apporach by formulating something like the most reasonable case where capturing the innumerable aspects of the topic is actually on point. One could think that 3, 3.1, $\pi$ and golden ratio would be perfectly legit options and questions of the form “Do you prefer your neighbour to be A far away or B far away?” would need to be answerable for all valid options and one option for conceiving it is for A and B to be arbitrary reals. With the “good-enough approximation” we don’t talk about being $\pi$ distance away because we don’t believe in truly trancendental distances. There is one distance a little beyond that and one little short of that and claims need to be about those.

[LGS]

Well, technically you can still restrict to finite-support probability distributions even if your outcome space is infinite. So even if you allow all real numbers as distances (and have utilities for each), you can restrict your set of beliefs to have finite support at any given time (i.e. at one point in time you might believe the distance to be one of {3,3.1,pi} or any other finite set of reals, and you may pick any distribution over that finite set). This setup still avoids Paul’s paradoxes.

Having said that, I have trouble seeing why you’d need to do this for the specific case of distances. Computers already use float-point arithmetic (of finite precision) to estimate real numbers, and not much goes wrong there. So computers are already restricting the set of possible distances to a finite set.

[Slider]

Any gradualation is likely to not hit the exact distance on the spot. Then If I was faced to be in a situation where I could become sensitive to that distinction I would need to go from not having included it in the support to having inluded it in the support ie from zero probablity to non-zero probablity. This seems like a smell that things are not genuine comparable to the safety of avoiding unbounded utilities, so i am not sure whether it is an improvement.

Even computers can do symbolic manipulation where they can get exact results. They are not forced to numerically simulate everything. Determining the intersection of two lines can be done exactly in finite computation despite doing it by “brute force” point-for-point whether they are in the same location would call for more than numerable steps.

I have intuitiion/​introspective impression that there are objects like “distance is {3,3.1, between 3.2 and 3.3}” where the three categories are equiprobable and distances within “3.2 to 3.3″ are equiprobable to each other but that “3.2 to 3.3” is not made up of listable separate beliefs. (More realisticially they tend to not feel exactly equiprobable within the whole range).

[deleted]

[Pattern]

in my last post

Suggested link fix:

replace:

https://​​www.lesswrong.com/​​editPost?postId=hbmsW2k9DxED5Z4eJ&eventForm=false

with

https://​​www.lesswrong.com/​​posts/​​hbmsW2k9DxED5Z4eJ

[paulfchristiano]

Fixed, thanks!