Impossibility results for unbounded utilities

[paulfchristiano]

Some people think that they have unbounded utility functions. This isn’t necessarily crazy, but it presents serious challenges to conventional decision theory. I think it probably leads to abandoning probability itself as a representation of uncertainty (or at least any hope of basing decision theory on such probabilities). This may seem like a drastic response, but we are talking about some pretty drastic inconsistencies.

This result is closely related to standard impossibility results in infinite ethics. I assume it has appeared in the philosophy literature, but I couldn’t find it in the SEP entry on the St. Petersburg paradox so I’m posting it here. (Even if it’s well known, I want something simple to link to.)

(ETA: this argument is extremely similar to Beckstead and Thomas’ argument against Recklessness in A paradox for tiny probabilities and enormous values. The main difference is that they use transitivity +”recklessness” to get a contradiction whereas I argue directly from “non-timidity.” I also end up violating a dominance principle which seems even more surprising to violate, but at this point it’s kind of like splitting hairs. I give a slightly stronger set of arguments in Better impossibility results for unbounded utilities.)

Weak version

We’ll think of preferences as relations $<$ over probability distributions over some implicit space of outcomes $\Omega$ (and we’ll identify outcomes with the constant probability distribution). We’ll show that there is no relation $<$ which satisfies three properties: Antisymmetry, Unbounded Utilities, and Dominance.

Note that we assume nothing about the existence of an underlying utility function. We don’t even assume that the preference relation is complete or transitive.

The properties

Antisymmetry: It’s never the case that both $A<B$ and $B<A$.

Unbounded Utilities: there is an infinite sequence of outcomes $X_1,X_2,X_4,X_8,\dots$ each “more than twice as good” as the last.^[1] More formally, there exists an outcome $X_0$ such that:

• $X_{2^k}<\frac{1}{2}{X}_{2^{k+1}}+\frac{1}{2}{X}_0$ for every $k$.

• $X_0<\frac{1}{2}{X}_1+\frac{1}{2}{X}_0$ ^[2]

That is, $X_1$ is not as good as a $\frac{1}{2}$ chance of $X_2$, which is not as good as a $\frac{1}{4}$ chance of $X_4$, which is not as good as a $\frac{1}{8}$ chance of $X_8$… This is nearly the weakest possible version of unbounded utilities.^[3]

Dominance: let $A_0,A_1,\dots$ and $B_0,B_1,\dots$ be sequences of lotteries, and $p_0,p_1,\dots$ be a sequence of probabilities that sum to 1. If $A_i<B_i$ for all $i$, then ${\sum }_i{p}_i{A}_i<{\sum }_i{p}_i{B}_i$.

Inconsistency proof

Consider the lottery $X_{\infty }=\frac{1}{2}{X}_0+\frac{1}{4}{X}_1+\frac{1}{8}{X}_2+\frac{1}{16}{X}_4+\dots$

We can write $X_{\infty }$ as a mixture:

• $X_{\infty }=\frac{1}{2}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_1)+\frac{1}{4}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_2)+\frac{1}{8}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_4)+\dots$

By definition $X_0<\frac{1}{2}{X}_0+\frac{1}{2}{X}_1$. And for each $k$, Unbounded Utilities implies that $X_{2^k}<\frac{1}{2}{X}_0+\frac{1}{2}{X}_{2^{k+1}}$. Thus Dominance implies $X_{\infty }<X_{\infty }$, contradicting Antisymmetry.

How to avoid the paradox?

By far the easiest way out is to reject Unbounded Utilities. But that’s just a statement about our preferences, so it’s not clear we get to “reject” it.

Another common way out is to assume that any two “infinitely good” outcomes are incomparable, and therefore to reject Dominance.^[4] This results in being indifferent to receiving $1 in every world (if the expectation is already infinite), or doubling the probability of all good worlds, which seems pretty unsatisfying.

Another option is to simply ignore small probabilities, which again leads to rejecting even the finite version of Dominance—sometimes when you mix together lotteries something will fall below the “ignore it” threshold leading the direction of your preference to reverse. I think this is pretty bizarre behavior, and in general ignoring small probabilities is much less appealing than rejecting Unbounded Utilities.

All of these options seem pretty bad to me. But in the next section, we’ll show that if the unbounded utilities are symmetric—if there are both arbitrarily good and arbitrarily bad outcomes—then things get even worse.

Strong version

I expect this argument is also known in the literature; but I don’t feel like people around LW usually grapple with exactly how bad it gets.

In this section we’ll show there is no relation $<$ which satisfies three properties: Antisymmetry, Symmetric Unbounded Utilities, and Weak Dominance.

(ETA: actually I think that even with only positive utilities you already violate something very close to Weak Dominance, which Beckstead and Thomas call Prospect-Outcome dominance. I find this version of Weak Dominance slightly more compelling, but Symmetric Unbounded Utilities is a much stronger assumption than Unbounded Utilities or non-Timidity, so it’s probably worth being aware of both versions. In a footnote^[5] I also define an even weaker dominance principle that we are forced to violate.)

The properties

Antisymmetry: It’s never the case that both $A>B$ and $B>A$.

Symmetric Unbounded Utilities. There is an infinite sequence of outcomes $X_1,X_{-2},X_4,X_{-8},\dots$ each of which is “more than twice as important” as the last but with opposite sign. More formally, there is an outcome $X_0$ such that:

• $X_0<X_1$

• For every even $k$ : $\frac{1}{3}{X}_{-2^{k+1}}+\frac{2}{3}{X}_{2^k}<X_0$

• For every odd $k$ : $\frac{1}{3}{X}_{2^{k+1}}+\frac{2}{3}{X}_{-2^k}>X_0$

That is, a certainty of $X_1$ is outweighed by a $\frac{1}{2}$ chance of $X_{-2}$, which is outweighed by a $\frac{1}{4}$ chance of $X_4$, which is outweighed by a $\frac{1}{8}$ chance of $X_{-8}$….

Weak Dominance.^[5] For any outcome $X$, any sequence of lotteries $B_0,B_1,\dots$, and any sequence of probabilities $p_0,p_1,\dots$ that sum to 1:

• If $X<B_i$ for every $i$, then $X<\sum p_i{B}_i$.

• If $X>B_i$ for every $i$, then $X>\sum p_i{B}_i$.

Inconsistency proof

Now consider the lottery $X_{\pm \infty }=\frac{1}{2}{X}_1+\frac{1}{4}{X}_{-2}+\frac{1}{8}{X}_4+\frac{1}{16}{X}_{-8}+\frac{1}{32}{X}_{16}+\dots$

We can write $X_{\pm \infty }$ as the mixture:

• $\frac{3}{4}(\frac{2}{3}{X}_1+\frac{1}{3}{X}_{-2})+\frac{3}{16}(\frac{2}{3}{X}_4+\frac{1}{3}{X}_{-8})+\frac{3}{64}(\frac{2}{3}{X}_{16}+\frac{1}{3}{X}_{-32})\dots$

By Unbounded Utilities each of these terms is $<X_0$. So by Weak Dominance, $X_{\pm \infty }<X_0.$

But we can also write $X_{\pm \infty }$ as the mixture:

• $\frac{1}{2}{X}_1+\frac{3}{8}(\frac{2}{3}{X}_{-2}+\frac{1}{3}{X}_4)+\frac{3}{32}(\frac{2}{3}{X}_{-8}+\frac{1}{3}{X}_{16})+\dots$

By Unbounded Utilities each of these terms is $>X_0$. So by Weak Dominance $X_{\pm \infty }>X_0$. This contradicts Antisymmetry.

Now what?

As usual, the easiest way out is to abandon Unbounded Utilities. But if that’s just the way you feel about extreme outcomes, then you’re in a sticky situation.

You could allow for unbounded utilities as long as they only go in one direction. For example, you might be open to the possibility of arbitrarily bad outcomes but not the possibility of arbitrarily good outcomes.^[6] But the asymmetric version of unbounded utilities doesn’t seem very intuitively appealing, and you still have to give up the ability to compare any two infinitely good outcomes (violating Dominance).

People like talking about extensions of the real numbers, but those don’t help you avoid any of the contradictions above. For example, if you want to extend $<$ to a preference order over hyperreal lotteries, it’s just even harder for it to be consistent.

Giving up on Weak Dominance seems pretty drastic. At that point you are talking about probability distributions, but I don’t think you’re really using them for decision theory—it’s hard to think of a more fundamental axiom to violate. Other than Antisymmetry, which is your other option.

At this point I think the most appealing option, for someone committed to unbounded utilities, is actually much more drastic: I think you should give up on probabilities as an abstraction for describing uncertainty, and should not try to have a preference relation over lotteries at all.^[7] There are no ontologically fundamental lotteries to decide between, so this isn’t necessarily so bad. Instead you can go back to talking directly about preferences over uncertain states of affairs, and build a totally different kind of machinery to understand or analyze those preferences.

ETA: replacing dominance

Since writing the above I’ve become more sympathetic to violations of Dominance and even Weak Dominance—it would be pretty jarring to give up on them, but I can at least imagine it. I still think violating “Very Weak Dominance”^[5] is pretty bad, but I don’t think it captures the full weirdness of the situation.

So in this section I’ll try to replace Weak Dominance by a principle I find even more robust: if I am indifferent between $X$ and any of the lotteries $A_i$, then I’m also indifferent between X and any mixture of the lotteries $A_i$. This isn’t strictly weaker than Weak Dominance, but violating it feels even weirder to me. At any rate, it’s another fairly strong impossibility result constraining unbounded utilities.

The properties

We’ll work with a relation $\le$ over lotteries. We write $A=B$ if both $A\le B$ and $B\le A$. We write $A<B$ if $A\le B$ but not $A=B$. We’ll show that $\le$ can’t satisfy four properties: Transitivity, Intermediate mixtures, Continuous symmetric unbounded utilities, and Indifference to homogeneous mixtures.

Intermediate mixtures. If $A<B$, then $A<\frac{1}{2}A+\frac{1}{2}B<B$.

Transitivity. If $A\le B$ and $B\le C$ then $A\le C$.

Continuous symmetric unbounded utilities. There is an infinite sequence of lotteries $X_1,X_{-2},X_4,X_{-8},\dots$ each of which is “exactly twice as important” as the last but with opposite sign. More formally, there is an outcome $X_0$ such that:

• $X_0<X_1$

• For every even $k$ : $\frac{1}{3}{X}_{-2^{k+1}}+\frac{2}{3}{X}_{2^k}=X_0$

• For every odd $k$ : $\frac{1}{3}{X}_{2^{k+1}}+\frac{2}{3}{X}_{-2^k}=X_0$

That is, a certainty of $X_1$ is exactly offset by a $\frac{1}{2}$ chance of $X_{-2}$, which is exactly offset by a $\frac{1}{4}$ chance of $X_4$, which is exactly offset by a $\frac{1}{8}$ chance of $X_{-8}$….

Intuitively, this principle is kind of like symmetric unbounded utilities, but we assume that it’s possible to dial down each of the outcomes in the sequence (perhaps by mixing it with $X_0$) until the inequalities become exact equalities.

Homogeneous mixtures. Let $X$ be an outcome, $A_0,A_1,A_2,\dots$, a sequence of lotteries, and $p_0,p_1,p_2,\dots$ be a sequence of probabilities summing to 1. If $A_i=X$ for all $i$, then $\sum p_i{A}_i=X$.

Inconsistency proof

Consider the lottery $X_{\pm \infty }=\frac{1}{2}{X}_1+\frac{1}{4}{X}_{-2}+\frac{1}{8}{X}_4+\frac{1}{16}{X}_{-8}+\frac{1}{32}{X}_{16}+\dots$

We can write $X_{\pm \infty }$ as the mixture:

• $\frac{3}{4}(\frac{2}{3}{X}_1+\frac{1}{3}{X}_{-2})+\frac{3}{16}(\frac{2}{3}{X}_4+\frac{1}{3}{X}_{-8})+\frac{3}{64}(\frac{2}{3}{X}_{16}+\frac{1}{3}{X}_{-32})\dots$

By Unbounded Utilities each of these terms is $=X_0$. So by homogeneous mixtures, $X_{\pm \infty }=X_0.$

But we can also write $X_{\pm \infty }$ as the mixture:

• $\frac{1}{2}{X}_1+\frac{3}{8}(\frac{2}{3}{X}_{-2}+\frac{1}{3}{X}_4)+\frac{3}{32}(\frac{2}{3}{X}_{-8}+\frac{1}{3}{X}_{16})+\dots$

By Unbounded Utilities each of these terms other than the first is $=X_0$. So by Homogenous Mixtures, the combination of all terms other than the first is $=X_0$. Together with the fact that $X_1>X_0$, Intermediate Mixtures and Transitivity imply $X_{\pm \infty }>X_0$. But that contradicts $X_{\pm \infty }=X_0$.

1. ^

   Note that we could replace “more than twice as good” with “at least 0.00001% better” and obtain exactly the same result. You may find this modified version of the principle more appealing, and it is closer to non-timidity as defined in Beckstead and Thomas. Note that the modified principle implies the original by applying transitivity 100000 times, but you don’t actually need to apply transitivity to get a contradiction, you can just apply Dominance to a different mixture.

2. ^

   You may wonder why we don’t just write $X_0<X_1$. If we did this, we’d need to introduce an additional assumption that if $A<B$, $pA+(1-p)X_0<pB+(1-p)X_0$. This would be fine, but it seemed nicer to save some symbols and make a slightly weaker assumption.

3. ^

   The only plausibly-weaker definition I see is to say that there are outcomes $X_0<X_1$ and an infinite sequence $X_{>2},X_{>3},...$ such that for all $n$: $\frac{1}{n}{X}_{>n}+(1-\frac{1}{n})X_0>X_1$. If we replaced the $>$ with $=$ then this would be stronger than our version, but with the inequality it’s not actually sufficient for a paradox.
   
   To see this, consider a universe with three outcomes $X_0,X_1,X_{\infty }$ and a preference order $<$ that always prefers lotteries with higher probability of $X_{\infty }$ and breaks ties using by preferring a higher probability of $X_1$. This satisfies all of our other properties. It satisfies the weaker version of the axiom by taking $X_{>n}=X_{\infty }$ for all $n$, and it wouldn’t be crazy to say that it has “unbounded” utilities.

4. ^

   For realistic agents who think unbounded utilities are possible, it seems like they should assign positive probability to encountering a St. Petersburg paradox such that all decisions have infinite expected utility. So this is quite a drastic thing to give up on. See also: Pascal’s mugging.

5. ^

   I find this principle pretty solid, but it’s worth noting that the same inconsistency proof would work for the even weaker “Very Weak Dominance”: for any pair of outcomes with $X_0<X_1$, and any sequence of lotteries $B_i$ each strictly better than $X_1$, any mixture of the $B_i$ should at least be strictly better than $X_0$!

6. ^

   Technically you can also violate Symmetric Unbalanced Utility while having both arbitrarily good and arbitrarily bad outcomes, as long as those outcomes aren’t comparable to one another. For example, suppose that worlds have a real-valued amount of suffering and a real-valued amount of pleasure. Then we could have a lexical preference for minimizing expected suffering (considering all worlds with infinite expected suffering as incomparable), and try to maximize pleasure only as a tie-breaker (considering all worlds with infinite expected pleasure as incomparable).

7. ^

   Instead you could keep probabilities but abandon infinite probability distributions. But at this point I’m not exactly sure what unbounded utilities means—if each decision involves only finitely many outcomes, then in what sense do all the other outcomes exist? Perhaps I may face infinitely many possible decisions, but each involves only finitely many outcomes? But then what am I to make of my parent’s decisions while raising me, which affected my behavior in each of those infinitely many possible decisions? It seems like they face an infinite mixture of possible outcomes. Overall, it seems to me like giving up on infinitely big probability distributions implies giving up on the spirit of unbounded utilities, or else going down an even stranger road.

[niplav]

I really like this post because it directly clarified my position on ethics, namely making me abandon unbounded utilities. I want to give this post a Δ and +4 for doing that, and for being clearly written and fairly short.

[Scott Garrabrant]

I am not a fan of unbounded utilities, but it is worth noting that most (all?) the problems with unbounded utilties are actually a problem with utility functions that are not integrable with respect to your probabilities. It feels basically okay to me to have unbounded utilities as long as extremely good/​bad events are also sufficiently unlikely.

The space of allowable probability functions that go with an unbounded utility can still be closed under finite mixtures and conditioning on positive probability events.

Indeed, if you think of utility functions as coming from VNM, and you a space of lotteries closed under finite mixtures but not arbitrary mixtures, I think there are VNM preferences that can only correspond to unbounded utility functions, and the space of lotteries is such that you can’t make St. Petersburg paradoxes. (I am guessing, I didn’t check this.)

[davidad]

1. I strongly agree that the key problem with St. Petersburg (and Pasadena) paradoxes is utility not being integrable with respect to the lotteries/​probabilities. Non-integrability is precisely what makes 𝔼U undefined (as a real number), whereas unboundedness of U alone does not.

2. However, it’s also worth pointing out that the space of functions which are guaranteed to be integrable with respect to any probability measure is exactly the space of bounded (measurable) functions. So if one wants to save utilities’ unboundedness by arguing from integrability, that requires accepting some constraints on one’s beliefs (e.g., that they be finitely supported). If one doesn’t want to accept any constraints on beliefs, then accepting a boundedness constraint on utility looks like a very natural alternative.

3. I agree with your last paragraph. If the state space of the world is ℝ, and the utility function is the identity function, then the induced preferences over finitely-supported lotteries can only be represented by unbounded utility functions, but are also consistent and closed under finite mixtures.

4. Finite support feels like a really harsh constraint on beliefs. I wonder if there are some other natural ways to constrain probability measures and utility functions. For example, if we have a topology on our state-space, we can require that our beliefs be compactly supported and our utilities be continuous. “Compactly supported” is way less strict than “finitely supported,” and “continuous” feels more natural than “bounded.” What are some other pairs of “compromise” conditions such that any permissible utility function is integrable with respect to any permissible belief distribution? (Perhaps it would be nice to have one that allows Gaussian beliefs, say, which are neither finitely nor compactly supported.)

[Scott Garrabrant]

Note that if $P$ dominates $Q$ in the sense that there is a $c>0$ such that for all events $E$, $P\left(E\right)>c\cdot Q\left(E\right)$, $U$ is integrable wrt $P$, then I think $U$ is integrable wrt $Q$. I propose the space of all probability distribution dominated by a given distribution $P$.

Conveniently, if we move to semi-measures, we can take P to be the universal semi-measure. I think we can have our space of utility functions be anything integrable WRT the universal semi-measure, and our space of probabilities be anything lower semi-computable, and everything will work out nicely.

[paulfchristiano]

I think bounded functions are the only computable functions that are integrable WRT the universal semi-measure. I think this is equivalent to de Blanc 2007?

The construction is just the obvious one: for any unbounded computable utility function, any universal semi-measure must assign reasonable probability to a St Petersburg game for that utility function (since we can construct a computable St Petersburg game by picking a utility randomly then looping over outcomes until we find one of at least the desired utility).

[AlexMennen]

Compact support still seems like an unreasonably strict constraint to me, not much less so than finite support. Compactness can be thought of as a topological generalization of finiteness, so, on a noncompact space, compact support means assigning probability 1 to a subset that’s infinitely tiny compared to its complement.

[Scott Garrabrant]

I observe that I probably miscommunicated. I think multiple people took me to be arguing for a space of lotteries with finite support. That is NOT what I meant. That is sufficient, but I meant something more general when I said “lotteries closed under finite mixtures” I did not mean there only finitely many atomic worlds in the lottery. I only meant that there is a space of lotteries, some of which maybe have infinite support if you want to think about atomic worlds, and for any finite set of lotteries, you can take a finite mixture of those lotteries to get a new lottery in the space. The space of lotteries has to be closed under finite mixtures for VNM to make sense, but the emphasis is on the fact that it is not closed under all possible countable mixtures, not that the mixtures have finite support.

[Charlie Steiner]

Hm, what would that last thing look like?

Like, I agree that you can have gambles closed under finite but not countable gambling and the math works. But it seems like reality is a countably-additive sort of a place. E.g. if these different outcomes of a lottery are physical states of some system, QM is going to tell you to take some infinite sums. I’m just generally having trouble getting a grasp on what the world (and our epistemic state re. the world) would look like for this finite gambles stuff to make sense.

[Scott Garrabrant]

Note that you can take infinite sums, without being able to take all possible infinite sums.

I suspect it looks like you have a prior distribution, and the allowable probability distributions are those that you can get to from this distribution using finitely many bits of evidence.

[deleted]

[So8res]

I think this argument is cool, and I appreciate how distilled it is.

Basically just repeating what Scott said but in my own tongue: this argument leaves open the option of denying that (epistemic) probabilities are closed under countable combination, and deploying some sort of “leverage penalty” that penalizes extremely high-utility outcomes as extremely unlikely a priori.

I agree with your note that the simplicitly prior doesn’t implement leverage penalties. I also note that I’m pretty uncertain myself about how to pull off leverage penalties correctly, assuming they’re a good idea (which isn’t clear to me).

I note further that the issue as I see it arises even when all utilities are finite, but some are (“mathematically”, not merely cosmically) large (where numbers like 10^100 are cosmically large, and numbers like 3^^^3 are mathematically large). Like, why are our actions not dominated by situations where the universe is mathematically large? When I introspect, it doesn’t quite feel like the answer is “because we’re certain it isn’t”, nor “because utility maxes out at the cosmological scale”, but rather something more like “how would you learn that there may or may not be 3^^^3 happy people with your choice as the fulcrum?” plus a sense that you should be suspicious that any given action is more likely to get 3^^^3 utility than any other (even in the presence of Pascall muggers) until you’ve got some sort of compelling account of how the universe ended up so large and you ended up being the fulcrum anyway. (Which, notably, starts to feel intertwined with my confusion about naturalistic priors, and I have at least a little hope that a good naturalistic prior would resolve the issue automatically.)

Or in other words, “can utilities be unbounded?” is a proxy war for “can utilities be mathematically large?”, with the “utilities must be bounded” resolution in the former corresponding (at least seemingly) to “utilities can be at most cosmically large” in the later. And while that may be the case, I don’t yet feel like I understand reasoning in the face of large utilities, and your argument does not dispell my confusion, and so I remain confused.

And, to be clear, I’m not saying that this problem seems intractible to me. There are various lines of attack that seem plausible from here. But I haven’t seen anyone providing the “cognitive recepits” from mapping out those lines of reasoning and deconfusing themselves about big utilities. For all I know, “utilities should be bounded (and furthermore, max utility should be at most cosmically large)” is the right answer. But I don’t confuse this guess for understanding.

[paulfchristiano]

TL;DR: I think that the discussion in this post is most relevant when we talk about the utility of whole universes. And for that purpose, I think a leverage penalty doesn’t make sense.

A leverage penalty seems more appropriate for saying something like “it’s very unlikely that my decisions would have such a giant impact,” but I don’t think that should be handled in the utility function or decision theory.

Instead, I’d say: if it’s possible to have “pivotal” decisions that affect 3^^^3 people, then it’s also possible to have 3^^^3 people in “normal” situations all making their separate (correlated) decisions, eating 3^^^3 sandwiches, and so the stakes of everything are similarly mathematically big.

plus a sense that you should be suspicious that any given action is more likely to get 3^^^3 utility than any other

I think that if utilities are large but bounded, then I feel like everything “adds up to normality”—if there is a way to get 3^^^3 utility, it seems like “maximize option value, figure out what’s going on, stay sane” is a reasonable bet for maximizing EV (e.g. by maximizing probability of the great outcome).

Intuitively, this also seems like what you should end up doing even if utilities are “infinite” (an expression that seems ~meaningless).

You can’t actually make these arguments go through for unbounded or infinite utilities, and part of the point is to observe that no arguments go through with unbounded/​infinite utilities because the entire procedure is screwed.

Or in other words, “can utilities be unbounded?” is a proxy war for “can utilities be mathematically large?”, with the “utilities must be bounded” resolution in the former corresponding (at least seemingly) to “utilities can be at most cosmically large” in the later.

I feel like in this discussion it’s not helpful to talk directly about magnitudes of utility functions, and to just talk directly about our questions about preferences, since that’s presumably the thing we have intuitions about. (I’d say that even if we thought utility functions definitely made sense in the regime where you have unbounded preferences, but it seems doubly true given that utility functions don’t seem very likely to be the right abstraction for unbounded preferences.)

deploying some sort of “leverage penalty” that penalizes extremely high-utility outcomes as extremely unlikely a priori.

This seems to put you in a strange position though: you are not only saying that high-value outcomes are unlikely, but that you have no preferences about them. That is, they aren’t merely impossible-in-reality, they are impossible-in-thought-experiments.

I personally feel like even if the thought experiments are impossible, they fairly clearly illustrate that some of our claims about our preferences can’t be right. To the extent that those claims come in large part from the appeal of certain clean mathematical machinery, I think the right first move is probably to become more unhappy about that machinery. To the extent that we have strong intuitions about non-timidity or unboundedness, then “become disillusioned with certain mathematical machinery” won’t help and we’ll just have to deal directly with sorting out our intuitions (but I think it’s fairly unlikely that journey involves discovering any philosophical insight that restores the appeal of the mathematical machinery).

“how would you learn that there may or may not be 3^^^3 happy people with your choice as the fulcrum?”

How would you learn that there may or may not be a 10^100 future people with our choices as the fulcrum? Why would the same process not generalize? (And if it may happen in the future but not now, is that 0 probability?)

To give my own account of the situation:

• It’s easy to come to believe that there are 3^^^3 potential people and that their welfare depends on a small number of pivotal events (through exactly the same kind of argument that leads us to believe that there are 10^100 potential people).

• Under those conditions, it’s at least plausible that there will inevitably be roughly 3^^^3 dreamers who come to that belief incorrectly (or who come to that belief correctly, but who incorrectly believe that they themselves are participating in the pivotal event). This involves some kind of necessary relationship between moral value and agency, which isn’t obvious but at least feels plausible. Note that most rationalists already bite this bullet, in that they think that the vast majority of individuals who think they are at the “hinge of history” are in simulations (and hence mistaken).

• Despite believing that, the prospect of “I am actually in the pivotal event” can easily dominate expected utility calculations in particular cases.

• That said, it’s still the case that a large universe contains many mistaken dreamers, so the costs they pay (in pursuit of the mistaken belief that they are participating in the pivotal event) will be comparable to the scale of consequences of the pivotal event. You end up with a non-fanatical outlook on the pivotal event itself, such that it will only dominate the EV if you have normal evidence that the current time is pivotal. (You get this kind of “adding up to normality” under much weaker assumptions than a strong leverage penalty.)

• But it still matters how much more we care about bigger universes—if the universe appears to be finite or small, should we assume that we are missing something?

My personal answer is that infinite universes don’t seem infinitely more important than finite universes, and that 2x bigger universes generally don’t seem 2x as important. (I tentatively feel that amongst reasonably-large universes, value is almost independent of size—while thinking that within any given universe 2x more flourishing is much closer to 2x as valuable.)

But this really seems like a brute question about preferences. And in this context, I don’t think a leverage penalty feels like a plausible way to resolve the confusion.

[So8res]

if it’s possible to have “pivotal” decisions that affect 3^^^3 people, then it’s also possible to have 3^^^3 people in “normal” situations all making their separate (correlated) decisions, eating 3^^^3 sandwiches, and so the stakes of everything are similarly mathematically big.

Agreed.

This seems to put you in a strange position though: you are not only saying that high-value outcomes are unlikely, but that you have no preferences about them. That is, they aren’t merely impossible-in-reality, they are impossible-in-thought-experiments.

Perhaps I’m being dense, but I don’t follow this point. If I deny that my epistemic probabilities are closed under countable weighted sums, and assert that the hypothesis “you can actually play a St. Petersburg game for n steps” is less likely than it is easy-to-describe (as n gets large), in what sense does that render me unable to consider St. Petersburg games in thought experiments?

How would you learn that there may or may not be a 10^100 future people with our choices as the fulcrum? Why would the same process not generalize? (And if it may happen in the future but not now, is that 0 probability?)

The same process generalizes.

My point was not “it’s especially hard to learn that there are 3^^^3 people with our choices as the fulcrum”. Rather, consider the person who says “but shouldn’t our choices be dominated by our current best guesses about what makes the universe seem most enormous, more or less regardless of how implausibly bad those best guesses seem?”. More concretely, perhaps they say “but shouldn’t we do whatever seems most plausibly likely to satisfy the simulator-gods, because if there are simulator gods and we do please them then we could get mathematically large amounts of utility, and this argument is bad but it’s not 1 in 3^^^3 bad, so.” One of my answers to this is “don’t worry about the 3^^^3 happy people until you believe yourself upstream of 3^^^3 happy people in the analogous fashion to how we currently think we’re upstream of 10^50 happy people”.

And for the record, I agree that “maximize option value, figure out what’s going on, stay sane” is another fine response. (As is “I think you have made an error in assessing your insane plan as having higher EV than business-as-usual”, which is perhaps one argument-step upstream of that.)

I don’t feel too confused about how to act in real life; I do feel somewhat confused about how to formally justify that sort of reasoning.

My personal answer is that infinite universes don’t seem infinitely more important than finite universes, and that 2x bigger universes generally don’t seem 2x as important. (I tentatively feel that amongst reasonably-large universes, value is almost independent of size—while thinking that within any given universe 2x more flourishing is much closer to 2x as valuable.)

That sounds like you’re asserting that the amount of possible flourishing limits to some maximum value (as, eg, the universe gets large enough to implement all possible reasonably-distinct combinations of flourishing civilizations)?

I’m sympathetic to this view. I’m not fully sold, of course. (Example confusion between me and that view: I have conflicting intuitions about whether running an extra identical copy of the same simulated happy people is ~useless or ~twice as good, and as such I’m uncertain about whether tiling copies of all combinations of flourishing civilizations is better in a way that doesn’t decay.)

While we’re listing guesses, a few of my other guesses include:

• Naturalism resolves the issue somehow. Like, perhaps the fact that you need to be embedded somewhere inside the world with a long St. Petersburg game drives its probability lower than the length of the sentence “a long St. Petersburg game” in a relevant way, and this phenomenon generalizes, or something. (Presumably this would have to come hand-in-hand with some sort of finitist philosophy, that denies that epistemic probabilities are closed under countable combination, due to your argument above.)

• There is a maximum utility, namely “however good the best arrangement of the entire mathematical multiverse could be”, and even if it does wind up being the case that the amount of flourishing you can get per-instantiation fails to converge as space increases, or even if it does turn out that instantiating all the flourishing n times is n times as good, there’s still some maximal number of instantiations that the multiverse is capable of supporting or something, and the maximum utility remains well-defined.

• The whole utility-function story is just borked. Like, we already know the concept is philosophically fraught. There’s plausibly a utility number, which describes how good the mathematical multiverse is, but the other multiverses we intuitively want to evaluate are counterfactual, and counterfactual mathematical multiverses are dubious above and beyond the already-dubious mathematical multiverse. Maybe once we’re deconfused about this whole affair, we’ll retreat to somewhere like “utility functions are a useful abstraction on local scales” while having some global theory of a pretty different character.

• Some sort of ultrafinitism wins the day, and once we figure out how to be suitably ultrafinitist, we don’t go around wanting countable combinations of epistemic probabilities or worrying too much about particularly big numbers. Like, such a resolution could have a flavor where “Nate’s utilities are unbounded” becomes the sort of thing that infinitists say about Nate, but not the sort of thing a properly operating ultrafinitist says about themselves, and things turn out to work for the ultrafinitists even if the infinitists say their utilities are unbounded or w/​e.

To be clear, I haven’t thought about this stuff all that much, and it’s quite plausible to me that someone is less confused than me here. (That said, most accounts that I’ve heard, as far as I’ve managed to understand them, sound less to me like they come from a place of understanding, and more like the speaker has prematurely committed to a resolution.)

[paulfchristiano]

One of my answers to this is “don’t worry about the 3^^^3 happy people until you believe yourself upstream of 3^^^3 happy people in the analogous fashion to how we currently think we’re upstream of 10^50 happy people”.

My point was that this doesn’t seem consistent with anything like a leverage penalty.

And for the record, I agree that “maximize option value, figure out what’s going on, stay sane” is another fine response.

My point was that we can say lots about which actions are more or less likely to generate 3^^^3 utility even without knowing how the universe got so large. (And then this appears to have relatively clear implications for our behavior today, e.g. by influencing our best guesses about the degree of moral convergence.)

That sounds like you’re asserting that the amount of possible flourishing limits to some maximum value (as, eg, the universe gets large enough to implement all possible reasonably-distinct combinations of flourishing civilizations)?

In terms of preferences, I’m just saying that it’s not the case that for every universe, there is another possible universe so much bigger that I care only 1% as much about what happens in the smaller universe. If you look at a 10^20 universe and the 10^30 universe that are equally simple, I’m like “I care about what happens in both of those universes. It’s possible I care about the 10^30 universe 2x as much, but it might be more like 1.000001x as much or 1x as much, and it’s not plausible I care 10^10 as much.” That means I care about each individual life less if it happens in a big universe.

This isn’t why I believe the view, but one way you might be able to better sympathize is by thinking: “There is another universe that is like the 10^20 universe but copied 10^10 times. That’s not that much more complex than the 10^20 universe. And in fact total observer counts were already dominated by copies of those universes that were tiled 3^^^3 times, and the description complexity difference between 3^^^3 and 10^10 x 3^^^3 are not very large.” Of course unbounded utilities don’t admit that kind of reasoning, because they don’t admit any kind of reasoning. And indeed, the fact that the expectations diverge seem very closely related to the exact reasoning you would care most about doing in order to actually assess the relative importance of different decisions, so I don’t think the infinity thing is a weird case, it seems absolutely central and I don’t even know how to talk about what the view should be if the infinites didn’t diverge.

I’m not very intuitively drawn to views like “count the distinct experiences,” and I think that in addition to being kind of unappealing those views also have some pretty crazy consequences (at least for all the concrete versions I can think of).

I basically agree that someone who has the opposite view—that for every universe there is a bigger universe that dwarfs its importance—has a more complicated philosophical question and I don’t know the answer. That said, I think it’s plausible they are in the same position as someone who has strong brute intuitions that A>B, B>C, and C>A for some concrete outcomes A, B, C—no amount of philosophical progress will help them get out of the inconsistency. I wouldn’t commit to that pessimistic view, but I’d give it maybe 50/​50---I don’t see any reason that there needs to be a satisfying resolution to this kind of paradox.

Perhaps I’m being dense, but I don’t follow this point. If I deny that my epistemic probabilities are closed under countable weighted sums, and assert that the hypothesis “you can actually play a St. Petersburg game for n steps” is less likely than it is easy-to-describe (as n gets large), in what sense does that render me unable to consider St. Petersburg games in thought experiments?

Do you have preferences over the possible outcomes of thought experiments? Does it feel intuitively like they should satisfy dominance principles? If so, it seems like it’s just as troubling that there are thought experiments. Analogously, if I had the strong intuition that A>B>C>A, and someone said “Ah but don’t worry, B could never happen in the real world!” I wouldn’t be like “Great that settles it, no longer feel confused+troubled.”

[So8res]

My point was that this doesn’t seem consistent with anything like a leverage penalty.

I’m not particulalry enthusiastic about “artificial leverage penalties” that manually penalize the hypothesis you can get 3^^^3 happy people by a factor of 1/​3^^^3 (and so insofar as that’s what you’re saying, I agree).

From my end, the core of my objection feels more like “you have an extra implicit assumption that lotteries are closed under countable combination, and I’m not sold on that.” The part where I go “and maybe some sufficiently naturalistic prior ends up thinking long St. Petersburg games are ultimately less likely than they are simple???” feels to me more like a parenthetical, and a wild guess about how the weakpoint in your argument could resolve.

(My guess is that you mean something more narrow and specific by “leverage penalty” than I did, and that me using those words caused confusion. I’m happy to retreat to a broader term, that includes things like “big gambles just turn out not to unbalance naturalistic reasoning when you’re doing it properly (eg. b/​c finding-yourself-in-the-universe correctly handles this sort of thing somehow)”, if you have one.)

(My guess is that part of the difference in framing in the above paragraphs, and in my original comment, is due to me updating in response to your comments, and retreating my position a bit. Thanks for the points that caused me to update somewhat!)

My point was that we can say lots about which actions are more or less likely to generate 3^^^3 utility even without knowing how the universe got so large.

I agree.

In terms of preferences, I’m just saying...

This seems like a fine guess to me. I don’t feel sold on it, but that could ofc be because you’ve resolved confusions that I have not. (The sort of thing that would persuade me would be you demonstrating at least as much mastery of my own confusions than I possess, and then walking me through the resolution. (Which I say for the purpose of being upfront about why I have not yet updated in favor of this view. In particular, it’s not a request. I’d be happy for more thoughts on it if they’re cheap and you find generating them to be fun, but don’t think this is terribly high-priority.))

That means I care about each individual life less if it happens in a big universe.

I indeed find this counter-intuitive. Hooray for flatly asserting things I might find counter-intuitive!

Let me know if you want me to flail in the direction of confusions that stand between me and what I understand to be your view. The super short version is something like “man, I’m not even sure whether logic or physics comes first, so I get off that train waaay before we get to the Tegmark IV logical multiverse”.

(Also, to be clear, I don’t find UDASSA particularly compelling, mainly b/​c of how confused I remain in light of it. Which I note in case you were thinking that the inferential gap you need to span stretches only to UDASSA-town.)

Do you have preferences over the possible outcomes of thought experiments? Does it feel intuitively like they should satisfy dominance principles? If so, it seems like it’s just as troubling that there are thought experiments.

You’ve lost me somewhere. Maybe try backing up a step or two? Why are we talking about thought experiments?

One of my best explicit hypotheses for what you’re saying is “it’s one thing to deny closure of epistemic probabiltiies under countable weighted combination in real life, and another to deny them in thought experiments; are you not concerned that denying them in thought experiments is troubling?”, but this doesn’t seem like a very likely argument for you to be making, and so I mostly suspect I’ve lost the thread.

(I stress again that, from my perspective, the heart of my objection is your implicit assumption that lotteries are closed under countable combination. If you’re trying to object to some other thing I said about leverage penalties, my guess is that I micommunicated my position (perhaps due to a poor choice of words) or shifted my position in response to your previous comments, and that our arguments are now desynched.)

---

Backing up to check whether I’m just missing something obvious, and trying to sharpen my current objection:

It seems to me that your argument contains a fourth, unlisted assumption, which is that lotteries are closed under countable combination. Do you agree? Am I being daft and missing that, like, some basic weak dominance assumption implies closure of lotteries under countable combination? Assuming I’m not being daft, do you agree that your argument sure seems to leave the door open for people who buy antisymmetry, dominance, and unbounded utilities, but reject countable combination of lotteries?

[paulfchristiano]

From my end, the core of my objection feels more like “you have an extra implicit assumption that lotteries are closed under countable combination, and I’m not sold on that.” [...] It seems to me that your argument contains a fourth, unlisted assumption, which is that lotteries are closed under countable combination. Do you agree?

My formal argument is even worse than that: I assume you have preferences over totally arbitrary probability distributions over outcomes!

I don’t think this is unlisted though—right at the beginning I said we were proving theorems about a preference ordering $<$ defined over the space of probability distributions over a space of outcomes $\Omega$. I absolutely think it’s plausible to reject that starting premise (and indeed I suggest that someone with “unbounded utilities” ought to reject this premise in an even more dramatic way).

If you’re trying to object to some other thing I said about leverage penalties, my guess is that I miscommunicated my position

It seems to me that our actual situation (i.e. my actual subjective distribution over possible worlds) is divergent in the same way as the St Petersburg lottery, at least with respect to quantities like expected # of happy people. So I’m less enthusiastic about talking about ways of restricting the space of probability distributions to avoid St Petersburg lotteries. This is some of what I’m getting at in the parent, and I now see that it may not be responsive to your view. But I’ll elaborate a bit anyway.

There are universes with populations of $3\uparrow \uparrow \uparrow 3$ that seem only $2^{1000}$ times less likely than our own. It would be very surprising and confusing to learn that not only am I wrong but this epistemic state ought to have been unreachable, that anyone must assign those universes probability at most 1/​$3\uparrow \uparrow \uparrow 3$. I’ve heard it argued that you should be confident that the world is giant, based on anthropic views like SIA, but I’ve never heard anyone seriously argue that you should be perfectly confident that the world isn’t giant.

If you agree with me that in fact our current epistemic state looks like a St Petersburg lottery with respect to # of people, then I hope you can sympathize with my lack of enthusiasm.

All that is to say: it may yet be that preferences are defined over a space of probability distributions small enough to evade the argument in the OP. But at that point it seems much more likely that preferences just aren’t defined over probability distributions at all—it seems odd to hold onto probability distributions as the object of preferences while restricting the space of probability distributions far enough that they appear to exclude our current situation.

You’ve lost me somewhere. Maybe try backing up a step or two? Why are we talking about thought experiments?

Suppose that you have some intuition that implies A > B > C > A.

At first you are worried that this intuition must be unreliable. But then you realize that actually B is impossible in reality, so consistency is restored.

I claim that you should be skeptical of the original intuition anyway. We have gotten some evidence that the intuition isn’t really tracking preferences in the way you might have hoped that it was—because if it were correctly tracking preferences it wouldn’t be inconsistent like that.

The fact that B can never come about in reality doesn’t really change the situation, you still would have expected consistently-correct intuitions to yield consistent answers.

(The only way I’d end up forgiving the intuition is if I thought it was correctly tracking the impossibility of B. But in this case I don’t think so. I’m pretty sure my intuition that you should be willing to take a 1% risk in order to double the size of the world isn’t tracking some deep fact that would make certain epistemic states inaccessible.)

(That all said, a mark against an intuition isn’t a reason to dismiss it outright, it’s just one mark against it.)

[So8res]

Ok, cool, I think I see where you’re coming from now.

I don’t think this is unlisted though …

Fair! To a large degree, I was just being daft. Thanks for the clarification.

It seems to me that our actual situation (i.e. my actual subjective distribution over possible worlds) is divergent in the same way as the St Petersburg lottery, at least with respect to quantities like expected # of happy people.

I think this is a good point, and I hadn’t had this thought quite this explicitly myself, and it shifts me a little. (Thanks!)

(I’m not terribly sold on this point myself, but I agree that it’s a crux of the matter, and I’m sympathetic.)

But at that point it seems much more likely that preferences just aren’t defined over probability distributions at all

This might be where we part ways? I’m not sure. A bunch of my guesses do kinda look like things you might describe as “preferences not being defined over probability distributions” (eg, “utility is a number, not a function”). But simultaneously, I feel solid in my ability to use probabliity distributions and utility functions in day-to-day reasoning problems after I’ve chunked the world into a small finite number of possible actions and corresponding outcomes, and I can see a bunch of reasons why this is a good way to reason, and whatever the better preference-formalism turns out to be, I expect it to act a lot like probability distributions and utility functions in the “local” situation after the reasoner has chunked the world.

Like, when someone comes to me and says “your small finite considerations in terms of actions and outcomes are super simplified, and everything goes nutso when we remove all the simplifications and take things to infinity, but don’t worry, sanity can be recovered so long as you (eg) care less about each individual life in a big universe than in a small universe”, then my response is “ok, well, maybe you removed the simplifications in the wrong way? or maybe you took limits in a bad way? or maybe utility is in fact bounded? or maybe this whole notion of big vs small universes was misguided?”

It looks to me like you’re arguing that one should either accept bounded utilities, or reject the probability/​utility factorization in normal circumstances, whereas to me it looks like there’s still a whole lot of flex (ex: ‘outcomes’ like “I come back from the store with milk” and “I come back from the store empty-handed” shouldn’t have been treated the same way as ‘outcomes’ like “Tegmark 3 multiverse branch A, which looks like B” and “Conway’s game of life with initial conditions X, which looks like Y”, and something was going wrong in our generalization from the everyday to the metaphysical, and we shouldn’t have been identifying outcomes with universes and expecting preferences to be a function of probability distributions on those universes, but thinking of “returning with milk” as an outcome is still fine).

And maybe you’d say that this is just conceding your point? That when we pass from everyday reasoning about questions like “is there milk at the store, or not?” to metaphysical reasoning like “Conway’s Life, or Tegmark 3?”, we should either give up on unbounded utilities, or give up on thinking of preferences as defined on probability distributions on outcomes? I more-or-less buy that phrasing, with the caveat that I am open to the weak-point being this whole idea that metaphysical universes are outcomes and that probabilities on outcome-collections that large are reasonable objects (rather than the weakpoint being the probablity/​utility factorization per se).

it seems odd to hold onto probability distributions as the object of preferences while restricting the space of probability distributions far enough that they appear to exclude our current situation

I agree that would be odd.

One response I have is similar to the above: I’m comfortable using probability distributions for stuff like “does the store have milk or not?” and less comfortable using them for stuff like “Conway’s Life or Tegmark 3?”, and wouldn’t be surprised if thinking of mathematical universes as “outcomes” was a Bad Plan and that this (or some other such philosophically fraught assumption) was the source of the madness.

Also, to say a bit more on why I’m not sold that the current situation is divergent in the St. Petersburg way wrt, eg, amount of Fun: if I imagine someone in Vegas offering me a St. Petersburg gamble, I imagine thinking through it and being like “nah, you’d run out of money too soon for this to be sufficiently high EV”. If you’re like “ok, but imagine that the world actually did look like it could run the gamble infinitely”, my gut sense is “wow, that seems real sus”. Maybe the source of the susness is that eventually it’s just not possible to get twice as much Fun. Or maybe it’s that nobody anywhere is ever in a physical position to reliably double the amount of Fun in the region that they’re able to affect. Or something.

And, I’m sympathetic to the objection “well, you surely shouldn’t assign probability less than <some vanishingly small but nonzero number> that you’re in such a situation!”. And maybe that’s true; it’s definitely on my list of guesses. But I don’t by any means feel forced into that corner. Like, maybe it turns out that the lightspeed limit in our universe is a hint about what sort of universes can be real at all (whatever the heck that turns out to mean), and an agent can’t ever face a St. Petersburgish choice in some suitably general way. Or something. I’m more trying to gesture at how wide the space of possibilities seems to me from my state of confusion, than to make specific counterproposals that I think are competitive.

(And again, I note that the reason I’m not updating (more) towards your apparently-narrower stance, is that I’m uncertain about whether you see a narrower space of possible resolutions on account of being less confused than I am, vs because you are making premature philosophical commitments.)

To be clear, I agree that you need to do something weirder than “outcomes are mathematical universes, preferences are defined on (probability distributions over) those” if you’re going to use unbounded utilities. And again, I note that “utility is bounded” is reasonably high on my list of guesses. But I’m just not all that enthusiastic about “outcomes are mathematical universes” in the first place, so \shrug.

The fact that B can never come about in reality doesn’t really change the situation, you still would have expected consistently-correct intuitions to yield consistent answers.

I think I understand what you’re saying about thought experiments, now. In my own tongue: even if you’ve convinced yourself that you can’t face a St. Petersburg gamble in real life, it still seems like St. Petersburg gambles form a perfectly lawful thought experiment, and it’s at least suspicious if your reasoning procedures would break down facing a perfectly lawful scenario (regardless of whether you happen to face it in fact).

I basically agree with this, and note that, insofar as my confusions resolve in the “unbounded utilities” direction, I expect some sort of account of metaphysical/​anthropic/​whatever reasoning that reveals St. Petersburg gambles (and suchlike) to be somehow ill-conceived or ill-typed. Like, in that world, what’s supposed to happen when someone is like “but imagine you’re offered a St. Petersburg bet” is roughly the same as what’s supposed to happen when someone’s like “but imagine a physically identical copy of you that lacks qualia”—you’re supposed to say “no”, and then be able to explain why.

(Or, well, you’re always supposed to say “no” to the gamble and be able to explain why, but what’s up for grabs is whether the “why” is “because utility is bounded”, or some other thing, where I at least am confused enough to still have some of my chips on “some other thing”.)

---

To be explicit, the way that my story continues to shift in response to what you’re saying, is an indication of continued updating & refinement of my position. Yay; thanks.

[paulfchristiano]

I expect it to act a lot like probability distributions and utility functions in the “local” situation after the reasoner has chunked the world.

I agree with this: (i) it feels true and would be surprising not to add up to normality, (ii) coherence theorems suggest that any preferences can be represented as probabilities+utilities in the case of finitely many outcomes.

“utility is a number, not a function”

This is my view as well, but you still need to handle the dependence on subjective uncertainty. I think the core thing at issue is whether that uncertainty is represented by a probability distribution (where utility is an expectation).

(Slightly less important: my most naive guess is that the utility number is itself represented as a sum over objects, and then we might use “utility function” to refer to the thing being summed.)

Also, to say a bit more on why I’m not sold that the current situation is divergent in the St. Petersburg way wrt, eg, amount of Fun...

I don’t mean that we face some small chance of encountering a St Petersburg lottery. I mean that when I actually think about the scale of the universe, and what I ought to believe about physics, I just immediately run into St Petersburg-style cases:

• It’s unclear whether we can have an extraordinarily long-lived civilization if we reduce entropy consumption to ~0 (e.g. by having a reversible civilization). That looks like at least 5% probability, and would suggest the number of happy lives is much more than ${10}^{100}$ times larger than I might have thought. So does it dominate the expectation?

• But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe. Maybe that happens with only 1% probability, but it corresponds to yet bigger civilization. So does that mean we should think that colonizing faster increases the value of the future by 1%, or by 100% since these possibilities are bigger and better and dominate the expectation?

• But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it (but a large fraction of them are nevertheless filled with other civilizations like ours). Perhaps that’s 20%. So it actually looks more likely than the “long-lived reversible civilization” and implies more total flourishing. And on those perspectives not going extinct is more important than going faster, for the usual reasons. So does that dominate the calculus instead?

• Perhaps rather than having a single set of physical constants, our universe runs every possible set. If that’s 5%, and could stack on top of any of the above while implying another factor of ${10}^{100}$ of scale. And if the standard model had no magic constants maybe this possibility would be 1% instead of 5%. So should I updated by a factor of 5 that we won’t discover that the standard model has fewer magic constants, because then “continuum of possible copies of our universe running in parallel” has only 1% chance instead of 5%?

• Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant ${10}^{100}$ more people, then it starts to seem like even less of a further ask to assume that there were actually ${10}^{1000}$ more people instead. So should we assume that multiple of those enhugening assumptions are true (since each one increases values by more than it decreases probability), or just take our favorite and then keep cranking up the numbers larger and larger (with each cranking being more probable than the last and hence more probable than adding a second enhugening assumption)?

Those are very naive physically statements of the possibilities, but the point is that it seems easy to imagine the possibility that populations could be vastly larger than we think “by default”, and many of those possibilities seem to have reasonable chances rather than being vanishingly unlikely. And at face value you might have thought those possibilities were actually action-relevant (e.g. the possibility of exponential returns to resources dominates the EV and means we should rush to colonize after all), but once you actually look at the whole menu, and see how the situation is just obviously paradoxical in every dimension, I think it’s pretty clear that you should cut off this line of thinking.

A bit more precisely: this situation is structurally identical to someone in a St Petersburg paradox shuffling around the outcomes and finding that they can justify arbitrary comparisons because everything has infinite EV and it’s easy to rewrite. That is, we can match each universe U with “U but with long-lived reversible civilizations,” and we find that the long-lived reversible civilizations dominate the calculus. Or we can match each universe U with “U but vast” and find the vast universes dominate the calculus. Or we can match “long-lived reversible civilizations” with “vast” and find that we can ignore long-lived reversible civilizations. It’s just like matching up the outcomes in the St Petersburg paradox in order to show that any outcome dominates itself.

The unbounded utility claim seems precisely like the claim that each of those less-likely-but-larger universes ought to dominate our concern, compared to the smaller-but-more-likely universe we expect by default. And that way of reasoning seems like it leads directly to these contradictions at the very first time you try to apply it to our situation (indeed, I think every time I’ve seen someone use this assumption in a substantive way it has immediately seemed to run into paradoxes, which are so severe that they mean they could as well have reached the opposite conclusion by superficially-equally-valid reasoning).

I totally believe you might end up with a very different way of handling big universes than “bounded utilities,” but I suspect it will also lead to the conclusion that “the plausible prospect of a big universe shouldn’t dominate our concern.” And I’d probably be fine with the result. Once you divorce unbounded utilities from the usual theory about how utilities work, and also divorce them from what currently seems like their main/​only implication, I expect I won’t have anything more than a semantic objection.

More specifically and less confidently, I do think there’s a pretty good chance that whatever theory you end up with will agree roughly with the way that I handle big universes—we’ll just use our real probabilities of each of these universes rather than focusing on the big ones in virtue of their bigness, and within each universe we’ll still prefer have larger flourishing populations. I do think that conclusion is fairly uncertain, but I tentatively think it’s more likely we’ll give up on the principle “a bigger civilization is nearly-linearly better within a given universe” than on the principle “a bigger universe is much less than linearly more important.”

And from a practical perspective, I’m not sure what interim theory you use to reason about these things. I suspect it’s mostly academic for you because e.g. you think alignment is a 99% risk of death instead of a 20% risk of death and hence very few other questions about the future matter. But if you ever did find yourself having to reason about humanity’s long-term future (e.g. to assess the value of extinction risk vs faster colonization, or the extent of moral convergence), then it seems like you should use an interim theory which isn’t fanatical about the possibility of big universes—because the fanatical theories just don’t work, and spit out inconsistent results if combined with our current framework. You can also interpret my argument as strongly objecting to the use of unbounded utilities in that interim framework.

[So8res]

This is my view as well,

(I, in fact, lifted it off of you, a number of years ago :-p)

but you still need to handle the dependence on subjective uncertainty.

Of course. (And noting that I am, perhaps, more openly confused about how to handle the subjective uncertainty than you are, given my confusions around things like logical uncertainty and whether difficult-to-normalize arithmetical expressions meaningfully denote numbers.)

Running through your examples:

It’s unclear whether we can have an extraordinarily long-lived civilization …

I agree. Separately, I note that I doubt total Fun is linear in how much compute is available to civilization; continuity with the past & satisfactory completion of narrative arcs started in the past is worth something, from which we deduce that wiping out civilization and replacing it with another different civilization of similar flourish and with 2x as much space to flourish in, is not 2x as good as leaving the original civilization alone. But I’m basically like “yep, whether we can get reversibly-computed Fun chugging away through the high-entropy phase of the universe seems like an empiricle question with cosmically large swings in utility associated therewith.”

But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe.

This seems fairly plausible to me! For instance, my best guess is that you can get more than 2x the Fun by computing two people interacting than by computing two individuals separately. (Although my best guess is also that this effect diminishes at scale, \shrug.)

By my lights, it sure would be nice to have more clarity on this stuff before needing to decide how much to rush our expansion. (Although, like, 1st world problems.)

But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it

Sure, this is pretty plausible, but (arguendo) it shouldn’t really be factoring into our action analysis, b/​c of the part where we can’t reach it. \shrug

Perhaps rather than having a single set of physical constants, our universe runs every possible set.

Sure. And again (arguendo) this doesn’t much matter to us b/​c the others are beyond our sphere of influence.

Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10^100 more people, then it starts to seem like even less of a further ask to assume that there were actually 10^1000 more people instead

I think this is where I get off the train (at least insofar as I entertain unbounded-utility hypotheses). Like, our ability to reversibly compute in the high-entropy regime is bounded by our error-correction capabilities, and we really start needing to upend modern physics as I understand it to make the numbers really huge. (Like, maybe 10^1000 is fine, but it’s gonna fall off a cliff at some point.)

---

I have a sense that I’m missing some deeper point you’re trying to make.

I also have a sense that… how to say… like, suppose someone argued “well, you don’t have 1/​∞ probability that “infinite utility” makes sense, so clearly you’ve got to take infinite utilities seriously”. My response would be something like “That seems mixed up to me. Like, on my current understanding, “infinite utility” is meaningless, it’s a confusion, and I just operate day-to-day without worrying about it. It’s not so much that my operating model assigns probability 0 to the proposition “infinite utilities are meaningful”, as that infinite utilities simply don’t fit into my operating model, they don’t make sense, they don’t typecheck. And separately, I’m not yet philosophically mature, and I can give you various meta-probabilities about what sorts of things will and won’t typecheck in my operating model tomorrow. And sure, I’m not 100% certain that we’ll never find a way to rescue the idea of infinite utilities. But that meta-uncertainty doesn’t bleed over into my operating model, and I’m not supposed to ram infinities into a place where they don’t fit just b/​c I might modify the type signatures tomorrow.”

When you bandy around plausible ways that the universe could be real large, it doesn’t look obviously divergent to me. Some of the bullets you’re handling are ones that I am just happy to bite, and others involve stuff that I’m not sure I’m even going to think will typecheck, once I understand wtf is going on. Like, just as I’m not compelled by “but you have more than 0% probability that ‘infinite utility’ is meaningful” (b/​c it’s mixing up the operating model and my philosophical immaturity), I’m not compelled by “but your operating model, which says that X, Y, and Z all typecheck, is badly divergent”. Yeah, sure, and maybe the resolution is that utilities are bounded, or maybe it’s that my operating model is too permissive on account of my philosophical immaturity. Philosophical immaturity can lead to an operating model that’s too permisive (cf. zombie arguments) just as easily as one that’s too strict.

Like… the nature of physical law keeps seeming to play games like “You have continua!! But you can’t do an arithmetic encoding. There’s infinite space!! But most of it is unreachable. Time goes on forever!! But most of it is high-entropy. You can do reversible computing to have Fun in a high-entropy universe!! But error accumulates.” And this could totally be a hint about how things that are real can’t help but avoid the truly large numbers (never mind the infinities), or something, I don’t know, I’m philisophically immature. But from my state of philosophical immaturity, it looks like this could totally still resolve in a “you were thinking about it wrong; the worst enhugening assumptions fail somehow to typecheck” sort of way.

---

Trying to figure out the point that you’re making that I’m missing, it sounds like you’re trying to say something like “Everyday reasoning at merely-cosmic scales already diverges, even without too much weird stuff. We already need to bound our utilities, when we shift from looking at the milk in the supermarket to looking at the stars in the sky (nevermind the rest of the mathematical multiverse, if there is such a thing).” Is that about right?

If so, I indeed do not yet buy it. Perhaps spell it out in more detail, for someone who’s suspicious of any appeals to large swaths of terrain that we can’t affect (eg, variants of this universe w/​ sufficiently different cosmological constants, at least in the regions where the locals aren’t thinking about us-in-particular); someone who buys reversible computing but is going to get suspicious when you try to drive the error rate to shockingly low lows?

To be clear, insofar as modern cosmic-scale reasoning diverges (without bringing in considerations that I consider suspicious and that I suspect I might later think belong in the ‘probably not meaningful (in the relevant way)’ bin), I do start to feel the vice grips on me, and I expect I’d give bounded utilities another look if I got there.

[Vanessa Kosoy]

A side note: IB physicalisms solves at least a large chunk of naturalism/​counterfactuals/​anthropics but is almost orthogonal to this entire issue (i.e. physicalist loss functions should still be bounded for the same reason cartesian loss functions should be bounded), so I’m pretty skeptical there’s anything in that direction. The only part which is somewhat relevant is: IB physicalists have loss functions that depend on which computations are running so two exact copies of the same thing definitely count as the same and not twice as much (except potentially in some indirect way, such as being involved together in a single more complex computation).

[So8res]

I am definitely entertaining the hypothesis that the solution to naturalism/​anthropics is in no way related to unbounded utilities. (From my perspective, IB physicalism looks like a guess that shows how this could be so, rather than something I know to be a solution, ofc. (And as I said to Paul, the observation that would update me in favor of it would be demonstrated mastery of, and unravelling of, my own related confusions.))

[Vanessa Kosoy]

In the parenthetical remark, are you talking about confusions related to Pascal-mugging-type thought experiments, or other confusions?

[So8res]

Those & others. I flailed towards a bunch of others in my thread w/​ Paul. Throwing out some taglines:

• “does logic or physics come first???”

• “does it even make sense to think of outcomes as being mathematical universes???”

• “should I even be willing to admit that the expression “3^^^3″ denotes a number before taking time proportional to at least log(3^^^3) to normalize it?”

• “is the thing I care about more like which-computations-physics-instantiates, or more like the-results-of-various-computations??? is there even a difference?”

• “how does the fact that larger quantum amplitudes correspond to more magical happening-ness relate to the question of how much more I should care about a simulation running on a computer with wires that are twice as thick???”

Note that these aren’t supposed to be particularly well-formed questions. (They’re more like handles for my own confusions.)

Note that I’m open to the hypothesis that you can resolve some but not others. From my own state of confusion, I’m not sure which issues are interwoven, and it’s plausible to me that you, from a state of greater clarity, can see independences that I cannot.

Note that I’m not asking for you to show me how IB physicalism chooses a consistent set of answers to some formal interpretations of my confusion-handles. That’s the sort of (non-trivial and virtuous!) feat that causes me to rate IB physicalism as a “plausible guess”.

In the specific case of IB physicalism, I’m like “maaaybe? I don’t yet see how to relate this Γ that you suggestively refer to as a ‘map from programs to results’ to a philosophical stance on computation and instantiation that I understand” and “I’m still not sold on the idea of handling non-realizability with inframeasures (on account of how I still feel confused about a bunch of things that inframeasures seem like a plausible guess for how to solve)” and etc.

Maybe at some point I’ll write more about the difference, in my accounting, between plausible guesses and solutions.

[Vanessa Kosoy]

Hmm… I could definitely say stuff about, what’s the IB physicalism take on those questions. But this would be what you specifically said you’re not asking me to do. So, from my perspective addressing your confusion seems like a completely illegible task atm. Maybe the explanation you alluded to in the last paragraph would help.

[So8res]

I’d be happy to read it if you’re so inclined and think the prompt would help you refine your own thoughts, but yeah, my anticipation is that it would mostly be updating my (already decent) probability that IB physicalism is a reasonable guess.

A few words on the sort of thing that would update me, in hopes of making it slightly more legible sooner rather than later/​never: there’s a difference between giving the correct answer to metaethics (“‘goodness’ refers to an objective (but complicated, and not objectively compelling) logical fact, which was physically shadowed by brains on account of the specifics of natural selection and the ancestral environment”), and the sort of argumentation that, like, walks someone from their confused state to the right answer (eg, Eliezer’s metaethics sequence). Like, the confused person is still in a state of “it seems to me that either morality must be objectively compelling, or nothing truly matters”, and telling them your favorite theory isn’t really engaging with their intuitions. Demonstrating that your favorite theory can give consistent answers to all their questions is something, it’s evidence that you have at least produced a plausible guess. But from their confused perspective, lots of people (including the nihilists, including the Bible-based moral realists) can confidently provide answers that seem superficially consistent.

The compelling thing, at least to me and my ilk, is the demonstration of mastery and the ability to build a path from the starting intuitions to the conclusion. In the case of a person confused about metaethics, this might correspond to the ability to deconstruct the “morality must be objectively compelling, or nothing truly matters” intuition, right in front of them, such that they can recognize all the pieces inside themselves, and with a flash of clarity see the knot they were tying themselves into. At which point you can help them untie the knot, and tug on the strings, and slowly work your way up to the answer.

(The metaethics sequence is, notably, a tad longer than the answer itself.)

(If I were to write this whole concept of solutions-vs-answers up properly, I’d attempt some dialogs that make the above more concrete and less metaphorical, but \shrug.)

In the case of IB physicalism (and IB more generally), I can see how it’s providing enough consistent answers that it counts as a plausible guess. But I don’t see how to operate it to resolve my pre-existing confusions. Like, we work with (infra)measures over ${\Sigma }^R\times \Phi$, and we say some fancy words about how ${\Sigma }^R$ is our “beliefs about the computations”, but as far as I’ve been able to make out this is just a neato formalism; I don’t know how to get to that endpoint by, like, starting from my own messy intuitions about when/​whether/​how physical processes reflect some logical procedure. I don’t know how to, like, look inside myself, and find confusions like “does logic or physics come first?” or “do I switch which algorithm I’m instantiating when I drink alcohol?”, and disassemble them into their component parts, and gain new distinctions that show me how the apparent conflicts weren’t true conflicts and all my previous intuitions were coming at things from slightly the wrong angle, and then shift angles and have a bunch of things click into place, and realize that the seeds of the answer were inside me all along, and that the answer is clearly that the universe isn’t really just a physical arrangement of particles (or a wavefunction thereon, w/​e), but one of those plus a mapping from syntax-trees to bits (here taking $|R|=2$). Or whatever the philosophy corresponding to “a hypothesis is a ${\Sigma }^R\times \Phi$” is supposed to be. Like, I understand that it’s a neat formalism that does cool math things, and I see how it can be operated to produce consistent answers to various philosophical questions, but that’s a long shot from seeing it solve the philosophical problems at hand. Or, to say it another way, answering my confusion handles consistently is not nearly enough to get me to take a theory philosophically seriously, like, it’s not enough to convince me that the universe actually has an assignment of syntax-trees to bits in addition to the physical state, which is what it looks to me like I’d need to believe if I actually took IB physicalism seriously.

[Vanessa Kosoy]

I don’t think I’m capable of writing something like the metaethics sequence about IB, that’s a job for someone else. My own way of evaluating philosophical claims is more like:

• Can we a build an elegant, coherent mathematical theory around the claim?

• Does the theory meet reasonable desiderata?

• Does the theory play nicely with other theories we have high confidence of?

• If there are compelling desiderata the theory doesn’t meet, can we show that meeting them is impossible?

For example, the way I understood objective morality is wrong was by (i) seeing that there’s a coherent theory of agents with any utility function whatsoever (ii) understanding that, in terms of the physical world, “Vanessa’s utility function” is more analogous to “coastline of Africa” than to “fundamental equations of physics”.

I agree that explaining why we have certain intuitions is a valuable source of evidence, but it’s entangled with messy details of human psychology that create a lot of noise. (Notice that I’m not saying you shouldn’t use intuition, obviously intuition is an irreplaceable core part of cognition. I’m saying that explaining intuition using models of the mind, while possible and desirable, is also made difficult by the messy complexity of human minds, which in particular introduces a lot of variables that vary between people.)

Also, I want to comment on your last tagline, just because it’s too tempting:

how does the fact that larger quantum amplitudes correspond to more magical happening-ness relate to the question of how much more I should care about a simulation running on a computer with wires that are twice as thick???

I haven’t written the proofs cleanly yet (because prioritizing other projects atm), but it seems that IB physicalism produces a rather elegant interpretation of QM. Many-worlds turns out to be false. The wavefunction is not “a thing that exists”. Instead, what exists is the outcomes of all possible measurements. The universe samples those outcomes from a distribution that is determined by two properties: (i) the marginal distribution of each measurement has to obey the Born rule (ii) the overall amount of computation done by the universe should be minimal. It follows that, outside of weird thought experiments (i.e. as long as decoherence applies), agents don’t get split into copies and quantum randomness is just ordinary randomness. (Another nice consequence is that Boltzmann brains don’t have qualia.)

[TAG]

What’s ordinary randomness?

[Vanessa Kosoy]

I think that this confusion results from failing to distinguish between your individual utility function and the “effective social utility function” (the result of cooperative bargaining between all individuals in a society). The individual utility function is bounded on a scale which is roughly comparable to Dunbar’s number^[1]. The effective social utility function is bounded on a scale comparable to the current size of humanity. When you conflate them, the current size of humanity seems like a strangely arbitrary parameter so you’re tempted to decide the utility function is unbounded.

The reason why distinguishing between those two is so hard, is because there are strong social incentives to conflate them, incentives which our instincts are honed to pick up on. Pretending to unconditionally follow social norms is a great way to seem trustworthy. When you combine it with an analytic mindset that’s inclined to reasoning with explicit utility functions, this self-deception takes the form of modeling your intrinsic preferences by utilitarianism.

Another complication is, larger universes tend to be more diverse and hence more interesting. But this also saturates somewhere (having e.g.${10}^{100}$ books to choose from is not noticeably better from having ${10}^{50}$ books to choose from).

---

1. ↩︎

   It seems plausible to me both for explaining how people behave in practice and in terms of evolutionary psychology.

[Slider]

The proof doesn’t run for me. The only way I know to be able to rearrange the terms in a infinite series is if the starting starting series converges and the resultant series converges. The series doesn’t fullfill the condition so I am not convinced the rewrite is a safe step.

I am a bit unsure about my maths so I am going to hyberbole the kind of flawed logic I read into the proof. Start with series that might not converge 1+1+1+1+1+1… (oh it indeed blatantly diverges) then split each term to have a non-effective addition (1+0)+(1+0)+(1+0)+(1+0)… . Blatantly disregard safety rules about paranthesis messing with series and just treat them as paranthesis that follow familiar rules 1+0+1+0+1+0+1+0+1… so 1+1+1+1… is not equal to itself. (unsafe step leads to non-sense)

With converging series it doesn’t matter whether we get “twice as fast” to the limit but the “rate of ascension” might matter to whatever analog a divergent series would have to a value.

[Maximum_Skull]

The correct condition for real numbers would be absolute convergence (otherwise the sum after rearrangement might become different and/​or infinite) but you are right: the series rearrangement is definitely illegal here.

[paulfchristiano]

But in the post I’m rearranging a series of probabilities, $\frac{1}{2},\frac{1}{4},\dots$ which is very legal. The fact that you can’t rearrange infinite sums is an intuitive reason to reject Weak Dominance, and then the question is how you feel about that.

[Maximum_Skull]

Those probabilities are multiplied by $X_i$s, which makes it more complicated.
If I try running it with $X$s being the real numbers (which is probably the most popular choice for utility measurement), the proof breaks down. If I, for example, allow negative utilities, I can rearrange the series from a divergent one into a convergent one and vice versa, trivially leading to a contradiction just from the fact that I am allowed to do weird things with infinite series, and not because of proposed axioms being contradictory.
EDIT: concisely, your axioms do not imply that the rearrangement should result in the same utility.

[LGS]

The rearrangement property you’re rejecting is basically what Paul is calling the “rules of probability” that he is considering rejecting.

If you have a probability distribution over infinitely (but countably) many probability distributions, each of which is of finite support, then it is in fact legal to “expand out” the probabilities to get one distribution over the underlying (countably infinite) domain. This is standard in probability theory, and it implies the rearrangement property that bothers you.

[Maximum_Skull]

Oh, thanks, I did not think about that! Now everything makes much more sense.

[paulfchristiano]

I’m not rearranging a sum of real numbers. I’m showing that no relationship $<$ over probability distributions satisfies a given dominance condition.

[Slider]

I am not familiar with the rules of lotteries and mixtures to know whether the mixture rewrite is valid or not. If the outcomes were for example money payouts then the operations carried out would be invalid. I would be surprised if somehow the rules for lotteries made this okay.

The bit where there is too much implicit steps for me is

Consider the lottery $X_{\infty }=\frac{1}{2}{X}_0+\frac{1}{4}{X}_1+\frac{1}{8}{X}_2+\frac{1}{16}{X}_4+\dots$

We can write $X_{\infty }$ as a mixture:

• $X_{\infty }=\frac{1}{2}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_1)+\frac{1}{4}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_2)+\frac{1}{8}(\frac{1}{2}{X}_0+\frac{1}{2}{X}_4)+\dots$

I would benefit from babystepping throught this process or atleast pointers what I need to learn to be convinced of this

[paulfchristiano]

I’m using the usual machinery of probability theory, and particularly countable additivity. It may be reasonable to give up on that, and so I think the biggest assumption I made at the beginning was that we were defining a probability distribution over arbitrary lotteries and working with the space of probability distributions.

A way to look at it is: the thing I’m taking sums over are the probabilities of possible outcomes. I’m never talking anywhere about utilities or cash payouts or anything else. The fact that I labeled some symbols $X_8$ does not mean that the real number 8 is involved anywhere.

But these sums over the probabilities of worlds are extremely convergent. I’m not doing any “rearrangement,” I’m just calculating ${\sum }_{k=n+1}^{\infty }\frac{1}{2^k}=\frac{1}{2^n}$.

[gjm]

So there are some missing axioms here, describing what happens when you construct lotteries out of other lotteries. Specifically, the rearranging step Slider asks about is not justified by the explicitly given axioms alone: it needs something along the lines of “if for each i we have a lottery ${\sum }_j{p}_{ij}{X}_j$, then the values of the lotteries ${\sum }_i{q}_i({\sum }_j{p}_{ij}{X}_j)$ and ${\sum }_j({\sum }_i{q}_i{p}_{ij})X_j$ are equal”.

(Your derivation only actually uses this in the special case where for each i only finitely many of the $p_{ij}$ are nonzero.)

You might want to say either that these two “different” lotteries have equal value, or else that they are in fact the same lottery.

In either case, it seems to me that someone might dispute the axiom in question (intuitively obvious though it seems, just like the others). You’ve chosen a notation for lotteries that makes an analogy with infinite series; if we take this seriously, we notice that this sort of rearrangement absolutely can change whether the series converges and to what value if so. How sure are you that rearranging lotteries is safer than rearranging sums of real numbers?

(The sums of the probabilities are extremely convergent, yes. But the probabilities are (formally) multiplying outcomes whose values we are supposing are correspondingly divergent. Again, I am not sure I want to assume that this sort of manipulation is safe.)

[paulfchristiano]

I’m handling lotteries as probability distributions over an outcome space $\Omega$, not as formal sums of outcomes.

To make things simple you can assume $\Omega$ is countable. Then a lottery $A$ assigns a real number $A\left(\omega \right)$ to each $\omega \in \Omega$, representing its probability under the lottery $A$, such that ${\sum }_{\omega \in \Omega }A\left(\omega \right)=1$. The sum $\sum p_i{A}_i$ is defined by $(\sum p_i{A}_i)\left(\omega \right)=\sum p_i{A}_i\left(\omega \right)$. And all these infinite sums of real numbers are in turn defined as the suprema of the finite sums which are easily seen to exist and to still sum to 1. (All of this is conventional notation.) Then ${\sum }_i{q}_i({\sum }_j{p}_{ij}{A}_j)$and ${\sum }_j({\sum }_i{q}_i{p}_{ij})A_j$ are exactly equal.

[gjm]

OK! But I still feel like there’s something being swept under the carpet here. And I think I’ve managed to put my finger on what’s bothering me.

There are various things we could require our agents to have preferences over, but I am not sure that probability distributions over outcomes is the best choice. (Even though I do agree that the things we want our agents to have preferences over have essentially the same probabilistic structure.)

A weaker assumptions we might make about agents’ preferences is that they are over possibly-uncertain situations, expressed in terms of the agent’s epistemic state.

And I don’t think “nested” possibly-uncertain-situations even exist. There is no such thing as assigning 50% probability to each of (1) assigning 50% probability to each of A and B, and (2) assigning 50% probability to each of A and C. There is such a thing as assigning 50% probability now to assigning those different probabilities in five minutes, and by the law of iterated expectations your final probabilities for A,B,C must then obey the distributive law, but the situations are still not literally the same, and I think that in divergent-utility situations we can’t assume that your preferences depend only on the final outcome distribution.

Another way to say this is that, given that the $A_i$ and $B_i$ are lotteries rather than actual outcomes and that combinations like $\sum p_i{A}_i$ mean something more complicated than they may initially look like they mean, the dominance axioms are less obvious than the notation makes them look, and even though there are no divergences in the sums-over-probabilities that arise when you do the calculations there are divergences in implied something-like-sums-over-weighted utilities, and in my formulation you really are having to rearrange outcomes as well as probabilities when you do the calculations.

[paulfchristiano]

I agree that in the real world you’d have something like “I’m uncertain about whether X or Y will happen, call it ⁵⁰⁄₅₀. If X happens, I’m ⁵⁰⁄₅₀ about whether A or B will happen. If Y happens, I’m ⁵⁰⁄₅₀ about whether B or C will happen.” And it’s not obvious that this should be the same as being ⁵⁰⁄₅₀ between B or X, and conditioned on X being ⁵⁰⁄₅₀ between A or C.

Having those two situations be different is kind of what I mean by giving up on probabilities—your preferences are no longer a function of the probability that outcomes occur, they are a more complicated function of your epistemic state, and so it’s not correct to summarize your epistemic state as a probability distribution over outcomes.

I don’t think this is totally crazy, but I think it’s worth recognizing it as a fairly drastic move.

[Bunthut]

Would a decision theory like this count as “giving up on probabilities” in the sense in which you mean it here?

[davidad]

To anyone who is still not convinced—that last move, ${\sum }_i{\sum }_j{q}_i{p}_{ij}{A}_j={\sum }_j{\sum }_i{q}_i{p}_{ij}{A}_j$, is justified by Tonelli’s theorem, merely because $q_i{p}_{ij}{A}_j\left(\omega \right)\ge 0$ (for all $i,j,\omega$).

[davidad]

The way I look at this is that objects like $\frac{1}{2}{X}_0+\frac{1}{2}{X}_1$ live in a function space like $X\to R_{\ge 0}$, specifically the subspace of that where the functions $f$ are integrable with respect to counting measure on $X$ and ${\sum }_{x\in X}f\left(x\right)=1$. In other words, objects like $f_1:=\frac{1}{2}{X}_0+\frac{1}{2}{X}_1$ are probability mass functions (pmf). $f_1\left(X_0\right)$ is $\frac{1}{2}$, and $f_1\left(X_1\right)$ is $\frac{1}{2}$, and $f_1$ of anything else is $0$. When we write what looks like an infinite series ${\lambda }_1{f}_1+{\lambda }_2{f}_2+\cdots$, what this really means is that we’re defining a new $f$ by pointwise infinite summation: $f\left(x\right):={\sum }_{i=1}^{\infty }{\lambda }_i{f}_i\left(x\right)$. So only each collection of terms that contains a given $X_k$ needs to form a convergent series in order for this new $f$ to be well-defined. And for it to equal another $f^{\prime }$, the convergent sums only need to be equal pointwise (for each $X_k$, $f\left(X_k\right)=f^{\prime }\left(X_k\right)$). In Paul’s proof above, the only $X_k$ for which the collection of terms containing it is even infinite is $X_0$. That’s the reason he’s “just calculating” that one sum.

[Slider]

The outcomes have the property that they are step-wise more than double the worth.

In $X_{\infty }=\frac{1}{2}{X}_0+\frac{1}{4}{X}_1+\frac{1}{8}{X}_2+\frac{1}{16}{X}_4+\dots$ the real part only halfs on each term. So as the series goes on each term gets bigger and bigger instead of smaller and smaller and smaller associated with convergent-like scenario. So it seems to me that even in isolation this is a divergent-like series.

[justinpombrio]

Here’s a concrete example. Start with a sum that converges to 0 (in fact every partial sum is 0):

0 + 0 + …

Regroup the terms a bit:

= (1 + −1) + (1 + −1) + …

= 1 + (-1 + 1) + (-1 + 1) + …

= 1 + 0 + 0 + …

and you get a sum that converges to 1 (in fact every partial sum is 1). I realize that the things you’re summing are probability distributions over outcomes and not real numbers, but do you have reason to believe that they’re better behaved than real numbers in infinite sums? I’m not immediately seeing how countable additivity helps. Sorry if that should be obvious.

[tailcalled]

Your argument doesn’t go through if you restrict yourself to infinite weighted averages with nonnegative weights.

[justinpombrio]

Aha. So if a sum of non-negative numbers converges, than any rearrangement of that sum will converge to the same number, but not so for sums of possibly-negative numbers?

Ok, another angle. If you take Christiano’s lottery:

$X_{\infty }=\frac{1}{2}{X}_0+\frac{1}{4}{X}_1+\frac{1}{8}{X}_2+\frac{1}{16}{X}_4...$

and map outcomes to their utilities, setting the utility of $X_0$ to 1, of $X_1$ to 2, etc., you get:

$1/2+1/2+1/2+1/2+...$

Looking at how the utility gets rearranged after the “we can write $X_{\infty }$ as a mixture” step, the first “1/​2″ term is getting “smeared” across the rest of the terms, giving:

$3/4+5/8+9/16+17/32+...$

which is a sequence of utilities that are pairwise higher. This is an essential part of the violation of Antisymmetry/​Unbounded/​Dominance. My intuition says that a strange thing happened when you rearranged the terms of the lottery, and maybe you shouldn’t do that.

Should there be another property, called “Rearrangement”?

Rearrangement: you may apply an infinite number of commutivity ($x+y=y+x$) and associativity ($(x+y)+z=x+(y+z)$) rewrites to a lottery.

(In contrast, I’m pretty sure you can’t get an Antisymmetry/​Unbounded/​Dominance violation by applying only finitely many commutivity and associativity rearrangements.)

I don’t actually have a sense of what “infinite lotteries, considered equivalent up to finite but not infinite rearrangements” look like. Maybe it’s not a sensible thing.

[Slider]

I am having trouble trying to translate between infinity-hiding style and explicit infinity style. My grievance with might be stupid.

$\frac{1}{2}{X}_0$

split X_0 into equal number parts to final form

$\frac{1}{2}(ϵX_0+ϵX_0+ϵX_0+...)$

move the scalar in

$\frac{1}{4}ϵX_0+\frac{1}{8}ϵX_0+\frac{1}{16}ϵX_0+...$

combine scalars

$\frac{ϵ}{4}{X}_0+\frac{ϵ}{8}{X}_0+\frac{ϵ}{16}{X}_0+...$

Take each of these separately to the rest of the original terms

$(\frac{ϵ}{4}{X}_0+\frac{1}{4}{X}_1)+(\frac{ϵ}{8}{X}_0+\frac{1}{8}{X}_2)+(\frac{ϵ}{16}{X}_0+\frac{1}{16})+...$

Combine scalars to try to hit closest to the target form

$\frac{1}{2}(\frac{ϵ}{2}{X}_0+\frac{1}{2}{X}_1)+\frac{1}{4}(\frac{ϵ}{2}{X}_0+\frac{1}{2}{X}_1)+\frac{1}{8}(\frac{ϵ}{2}{X}_0+\frac{1}{2}{X}_1)+...$

$\frac{ϵ}{2}{X}_0+\frac{1}{2}{X}_1$is then quite far from $\frac{1}{2}{X}_0+\frac{1}{2}{X}_1$

Within real precision a single term hasn’t moved much $\frac{ϵ}{2}{X}_0+\frac{1}{2}{X}_1\sim \frac{1}{2}{X}_1$

This suggests to me that somewhere there are “levels of calibration” that are mixing levels corresponding to members of different archimedean fields trying to intermingle here. Normally if one is allergic to infinity levels there are ways to dance around it /​ think about it in different terms. But I am not efficient in translating between them.

[Slider]

New attempt

$X_{\infty }=\frac{1}{2}{X}_0+\frac{1}{4}{X}_1+\frac{1}{8}{X}_2+\frac{1}{16}{X}_4+\dots$

I think I now agree that $X_0$ can be written as $\frac{1}{2}{X}_0+\frac{1}{4}{X}_0+\frac{1}{8}{X}_0...$

However this uses a “de novo” indexing and gets only to

$\frac{1}{2}$ ($\frac{1}{2}{X}_0+\frac{1}{4}{X}_0+\frac{1}{8}{X}_0...$)$+\frac{1}{4}{X}_1+\frac{1}{8}{X}_2+\frac{1}{16}{X}_4+\dots$

taking terms out form the inner thing crosses term lines for the outer summation which counts as “messing with indexing” in my intuition. The suspect move just maps them out one to one

$(\frac{1}{4}{X}_0+\frac{1}{4}{X}_1)+(\frac{1}{8}{X}_0+\frac{1}{8}{X}_2)+(\frac{1}{16}{X}_0+\frac{1}{16}{X}_4)+...$

But why is this the permitted way and could I jam the terms differently in say apply to every other term

$(\frac{1}{4}{X}_0+\frac{1}{4}{X}_1)+\left(\frac{1}{8}{X}_2\right)+(\frac{1}{8}{X}_0+\frac{1}{16}{X}_4)+\frac{1}{32}{X}_8+(\frac{1}{16}{X}_0+\frac{1}{64}{X}_{1}6)+...$

If I have $\left(\sum _{i=0}^a{x}_i\right)+\left(\sum _{j=0}^a{y}_j\right)$ I am more confident that they “index at the same rate” to make $\sum _{u=0}^c{x}_u+y_u$. However if I have $\left(\sum _i^a{x}_i\right)+\left(\sum _j^b{y}_j\right)$ I need more information about the relation of a and b to make sure that mixing them plays nicely. Say in the case of b=2a then it is not okay to think only of the terms when mixing.

[mikehawk]

I had the same initial reaction. I believe the logic of the proof is fine (it is similar to the Mazur swindle), basically because it it not operating on real numbers, but rather on mixtures of distributions.

The issue is more: why would you expect the dominance condition to hold in the first place? If you allow for unbounded utility functions, then you have to give it up anyway, for kind of trivial reasons. Consider two sequences Ai and Bi of gambles such that EA_i<EB_i and sum_i p_iEA_i and sum_i p_i EB_i both diverge. Does it follow that E(sum_i p_iA_i)< E(sum_i p_i B_i) ? Obviously not, since both quantities diverge. At best you can say <=. A bit more formally; in real analysis/​measure theory one works with the so-called extended real numbers, in which the value “infinity” is assigned to any divergent sum, with this value assumed to be defined by the algebraic property x<=infinity for any x. In particular, there is no x in the extended real numbers such that infinity<x. So at least in standard axiomatizations of measure theory, you cannot expect the strict dominance condition to hold in complete generality; you will have to make some kind of exception for infinite values. Similar considerations apply to the Intermediate Mixtures assumption.

[Slider]

With surreals I might have transfinite quantities that can reliably compare every which way despite both members being beyond a finite bound. For “tame” entities all kinds of nice properties are easy to get/​prove. The game of “how wild my entities can get while retaining a certain property” is a very different game. “These properties are impossible to get even for super-wild things” is even harder.

Mazur seems (atleast based on the wikipedia article) not to be a proof of certain things, so that warrants special interest whether the applicability conditions are met or not.

[Jalex Stark]

The sum we’re rearranging isn’t a sum of real numbers, it’s a sum in ${\ell }_1$. Ignoring details of what ${\ell }_1$ means… the two rearrangements give the same sum! So I don’t understand what your argument is.

Abstracting away the addition and working in an arbitrary topological space, the argument goes like this: $L=lim{x}_n=lim{y}_n$. For all $n,f\left(x_n\right)=0\text{and}f\left(y_n\right)=1.$ Therefore, f is not continuous (else 0 = 1).

[Slider]

if ${\ell }_1$ is something weird then I don’t neccesarily even know that x+y=y+x, it is not a given at all that rearrangement would be permissible.

In order to sensibly compare $lim{x}_n$ and $lim{y}_n$ it would be nice if they both existed and not be infinities.$L=lim{x}_n=lim{y}_n=\infty$ is not useful for transiting equalities between x and y.

[Jalex Stark]

L is not equal to infinity; that’s a type error. L is equal to ¹⁄₂ A_0 + ¹⁄₄ A_1 + ¹⁄₈ A_2 …

${\ell }_1$ is a bona fide vector space—addition behaves as you expect. The points are infinite sequences (x_i) such that ${\sum }_i|x_i|$ is finite. This sum is a norm and the space is Banach with respect to that norm.

Concretely, our interpretation is that x_i is the probability of being in world A_i.

A utility function is a linear functional, i.e. a map from points to real numbers such that the map commutes with addition. The space of continuous linear functionals on ${\ell }_1$ is ${\ell }_{\infty }$, which is the space of bounded sequences. A special case of this post is that unbounded linear functionals are not continuous. I say ‘special case’ because the class of “preference between points” is richer than the class of utility functions. You get a preference order from a utility function via “map to real numbers and use the order there.” The utility function framework e.g. forces every pair of worlds to be comparable, but the more general framework doesn’t require this—Paul’s theorem follows from weaker assumptions.

[Slider]

The presentation tries to deal with unbounded utilities. Assuming ${\sum }_i|x_1|$ to be finite exludes the target of investigation from the scope.

Supposedly there are multiple text input methods but atleast on the website I can highlight text and use a $f\left(x\right)$ button to get math rendering.

I don’t know enough about the fancy spaces whether a version where the norm can take on transfinite or infinidesimal values makes sense or that the elements are just sequences without a condition to converge. Either (real number times a outcome) is a type for which finiteness check doesn’t make sense or the allowable conversions from outcomes to real numbers forces the sum to be bigger than any real number.

[Jalex Stark]

Requiring ${\sum }_i|x_i|$ to be finite is just part of assuming the $x_i$ form a probability distribution over worlds. I think you’re confused about the type difference between the$A_i$ and the utility of $A_i$. (Where in the context of this post, the utility is just represented by an element of a poset.)

I’m not advocating for or making arguments about any fanciness related to infinitesimals or different infinite values or anything like that.

[Charlie Steiner]

Maybe I’m just a less charitable person—it seems very easy to me for someone to say the words “I have unbounded utility” without actually connecting any such referent to their decision-making process.

We can show that there’s a tension between that verbal statement and the basic machinery of decision-making, and also illustrates how the practical decision-making process people use every day doesn’t act like expected utilities diverge.

And I think the proper response to seeing something like this happen to you is definitely not to double down on the verbal statement that sounded good. It’s to stop and think very skeptically about whether this verbal statement fits with what you can actually ask of reality, and what you might want to ask for that you can actually get. (I’ve written too many posts about why it’s the wrong move to want an AI to “just maximize my utility function.” Saying that you want to be modeled as if you have unbounded utility [of this sort that lets you get divergent EV] is the same order of mistake.)

If you think people can make verbal statements that are “not up for grabs,” this probably seems like gross uncharitableness.

[paulfchristiano]

I can easily imagine people being mistaken about “would you prefer X or Y?” questions (either in the sense that their decisions would change on reflection, or their utterances aren’t reflective of what should be rightly called their preferences, or whatever).

That said, I also don’t think that it’s obvious that uncertainty should be represented as probabilities with preferences depending only the probability of outcomes.

That said, all things considered I feel like bounded utility functions are much more appealing than the other options. Mostly I wrote this post to help explain my serious skepticism about unbounded utility functions (and about how nonchalantly the prospect of unbounded utility functions is thrown around).

[Davidmanheim]

Just posting to say I’m strongly in agreement that unbounded utility functions aren’t viable—and we tried to deal with some of the issues raised by philosophers, with more or less success, in our paper here: https://​​philpapers.org/​​rec/​​MANWIT-6

[Davidmanheim]

This is basically what I tried to argue in my preprint with Anders on infinite value—to quote:

”We have been unfortunately unable to come up with a clear defense of the conceivability of infinities and infinitesimals used for decisionmaking, but will note a weak argument to illustrate the nonviable nature of the most common class of objection. The weak claim is that people can conceive of infinitesimals, as shown by the fact that there is a word for it, or that there is a mathematical formalism that describes it. But, we respond, this does not make a claim for the ability to conceive of a value any better than St. Anselm’s ontological proof of the existence of God. More comically, we can say that this makes the case approximately the same way someone might claim to understand infinity because they can draw an 8 sideways — it says nothing about their conception, much less the ability to make decisions on the basis of the infinite or infinitesimal value or probability. ”

[FireStormOOO]

This seems plausible to me for people who don’t live and breathe math but still think Expected Utility is a tool they can’t afford not to use. I would be surprised if the typical person, even here, picks up the subtlety with any of the infinite sums and weird implication of that on the first pass. I don’t think infinite sums (and their many pitfalls) are typically taught at all until Calc II, which is not even a graduation requirement for non-STEM undergrad degrees.

People also get a lot of mileage out of realizing that IRL most problems aren’t edge cases and even fewer are corner cases—rightly skipping most of the rigor that’s necessary when discussing philosophy and purposely seeking out weird edge cases.

Now if someone is actually well versed in the math and philosophizing and saying that understanding all the implications that’s an interesting discussion I want to read.

[Zach Stein-Perlman]

People like talking about extensions of the real numbers, but those don’t help you avoid any of the contradictions above. For example, if you want to extend < to a preference order over hyperreal lotteries, it’s just even harder for it to be consistent.

I’m a recent proponent of hyperreal utilities. I totally agree that hyperreals don’t solve issues with divergent /​ St. Petersburg-style lotteries. I just think hyperreals are perfect for describing and comparing potentially-infinite-utility universes, though not necessarily lotteries over those universes. (This doesn’t contradict Paul; I’m just clarifying.)

Separately, while cases like this do make it feel like we “should give up on probabilities as an abstraction for describing uncertainty,” this conclusion makes me feel quite nihilistic about decision-under-uncertainty; I will be utterly shocked if “a totally different kind of machinery to understand or analyze those preferences” is satisfactory.

[paulfchristiano]

My main concern is that unbounded utilities (and hence I assume also hyperreal utilities, unless you are just using them to express simple lexical preferences?) have a really hard time playing nicely with lotteries. But then if you aren’t describing preferences over lotteries, why do you want to have scalar utilities at all?

Separately, while cases like this do make it feel like we “should give up on probabilities as an abstraction for describing uncertainty,” this conclusion makes me feel quite nihilistic about decision-under-uncertainty; I will be utterly shocked if “a totally different kind of machinery to understand or analyze those preferences” is satisfactory.

I think I’d be less shocked by a totally different framework than probability, but I do agree that it looks kind of bleak. I would love to just reject unbounded utilities out of hand based on this kind of argument, and personally I don’t find unbounded utilities very appealing, but if someone swings that way I don’t feel like you can very well tell them to just change their preferences.

[abramdemski]

I’m also a fan of hyperreal probabilities/​utilities—not that I think humans (do/​should) use them, particularly, but that I think they’re not ruled out by appealing rationality principles.

My main concern is that unbounded utilities [...] have a really hard time playing nicely with lotteries. But then if you aren’t describing preferences over lotteries, why do you want to have scalar utilities at all?

I think the Jeffrey-Bolker axioms are more appealing than lottery-based elucidations of utility theory, and don’t have the sorts of problems you’re pointing to. In particular, you can just drop their version of the continuity axiom, which Jeffrey also feels is unmotivated.

Preferences can be represented by a probability distribution and an expected-utility distribution, but you can’t necessarily go the other way, from arbitrary probabilities and utilities to coherent preferences. So you can’t just define arbitrary lotteries like you do in the OP. The agent has to actually believe these to be possible.

So, rather than an impossibility result for unbounded utilities, I think you get that unbounded utilities are only consistent with specific beliefs.

EDIT: I still have to think about this more, but I think I misrepresented things a little. If we want expectations to be real-valued, but unbounded, then I think we can keep all the axioms including continuity, but the agent can’t believe in the possibility of lotteries corresponding to divergent sums. If we are OK with hyperreal values, then I think we drop continuity, and belief in lotteries with divergent sums are OK (but we still don’t need to deal with arbitrary lotteries, because that’s just not a very natural thing to do in the JB framework—we only want to work with what an agent believes is possible). However, it’s possible that your Dominance axiom gets violated. Intuitively, I don’t think this is a necessary thing (IE, it seems like we could add a Dominance-like axiom). In particular, your arguments don’t go through, since we don’t get to construct arbitrary lotteries; we only get to examine what the agent actually believes in and has preferences about. (So, to work with arbitrary lotteries, you have to add axioms asserting that events exist in the event-algebra corresponding to the desired lotteries.)

[davidad]

I’m really interested in this direction (largely because I’m already interested in pointless-topology/​geometric-logic approaches to world-modeling), but I have a couple concerns off the bat. (Maybe if I read more about Jeffrey-Bolker and the surrounding literature I can answer my own questions here, but I thought I’d ask now anyway.)

1. One of the neat things about the standard interpretation of vNM is that it gives me an algorithmic recipe for (a) eliciting my beliefs-and-preferences about simple events, and then (b) deducing uniquely-valid consequences about what my preferences about complex events have to be (by computing integrals), so I don’t have to think about the complex events directly. Is there anything analogous to this in the Jeffrey-Bolker world?

2. If there is, can I apply it to ordinary lotteries that I definitely believe are possible, like prediction-market payouts?

3. If so, what does it look like mechanistically when I try to apply it to a St. Petersburg lottery? Where exactly are the guardrails between ordinary lotteries and impossible lotteries, and how might I come to realize (from within the Jeffrey-Bolker framework) that I’m not allowed to believe that a St. Petersburg lottery is real?

[abramdemski]

Jeffrey does talk about this in his book! Denote the probability of an event as P(E), and the expected value as V(E). Now suppose we cut an event A into parts B and C. We must have that V(A) = V(B)P(B|A) + V(C)P(C|A). Using this, we can cut the world up into small events which we’re comfortable assigning values to, and then put things back together into expected values for larger events. Basically exactly what you’d do normally, but no events are distinguished as “outcomes” so you can start wherever you want.

The JB axioms don’t assume anything like countable additivity, so an event like “St. Petersburg Lottery” needs a valuation which is consistent (in the above-mentioned sense) with the value of all other events, but there isn’t (necessarily) a computation which tells you the value of the infinite sum, the way there is for finite sums. We can add axioms which constrain values in situations like that, to avoid absurd things; but since it isn’t clear how to evaluate infinite sums in general, it makes sense to keep those axioms weak.

I think of this as a true result: the value of infinite lotteries is subjective (even after we know how to value all their finite parts). It has a lot of coherence constraints, but not enough to fully pin down a value. This means we don’t have to worry about all the messy nonsense of trying to evaluate divergent sums.

However, I think if we assume that any sub-event of the st petersburg lottery in which we still have a chance of winning something has a positive value (which seems very reasonable), then we can prove that the total value of the lottery is not any real number, by splitting off more and more of the finite sub-lotteries and arguing that the total value must exceed each.

Where exactly are the guardrails between ordinary lotteries and impossible lotteries, and how might I come to realize (from within the Jeffrey-Bolker framework) that I’m not allowed to believe that a St. Petersburg lottery is real?

The continuity axiom is what stops you from having non-archimedean values (just like with VNM), so that’s where the buck stops. If our preferences respect continuity, then we have to choose between believing St-petersburg-like lotteries are possible, vs believing enough niceness axioms about the values of infinite lotteries to prove that St. Petersburg has a value greater than all the reals.

For me, it seems like a pretty obvious choice to reject continuity (as a rationality condition that must apply to all rational minds—it could very well apply to humans in a practical sense). Continuity seems poorly-motivated in the first place.

[Zach Stein-Perlman]

I think assigning real (or hyperreal) values to possible universes can give really aesthetic properties (edit: at least to me; probably much less aesthetic for those who “don’t find unbounded utilities very appealing”) that I’d roughly call “additivity” or “linearity,” like: if A and B are systems, U(universe containing A) + U(universe containing B) = U(universe containing A and B). (This assumes that value is local, or something, which seems reasonable.) Utilities contain more than just lexical-ordering information if we can use them to describe the utility of new possible universes.

Perhaps more importantly, real and hyperreal utilities seem to play nice with finite lotteries, which seems quite desirable (and quite enough reason to have scalar utilities), even though it isn’t as strong as we’d hope.

[paulfchristiano]

I think the dominance principle used in this post is too strong and relatively easy to deny. I think that the Better impossibility results for unbounded utilities are actually significantly better.

[Raemon]

This seems useful to be flagged as a review, so it shows up in some review UI later. Mind if I convert it?

(You can create reviews by clicking the Review button at the top of the post)

[niplav]

I guess in the context of the Review I consider the two posts as one.

[paulfchristiano]

I think that’s reasonable, this is the one with the discussion and it has a forward link, would be better to review them as a unit.

[Rafael Harth]

The examples in this post all work by not just having divergent sums but unbounded single elements. When I look at this, my immediate takeaway is that we need a model that includes time. Outcomes should be of the form $(x,t)\in R\times N$, where $t$ indicates at which timestep they happen.

You are then allowed to look at infinite sequences $(x_i,t_i),(x_{i+1},t_{i+1}),...$ iff the $t_i$ are strictly increasing. The sums $\sum x_i$ do not need to converge, but there does need to be a global bound for all individual utilities, i.e.$\exists B\in R$ that upper-bounds all $x_i$.

This avoids all the problems in this post and it seems much better than giving up on probabilities. In general, it seems to capture how people think about infinities; they generally don’t imagine a few moments of literally infinite bliss, but an eternity of some amount of well-being bounded away from zero.

This would then transform the problem from “unbounded utility functions are inconsistent” to “it’s hard to compare outcomes when divergent sums are involved”.

[paulfchristiano]

Yes, I think that having bounded single elements but infinitely big universes is potentially fine.

Though if the utilities of worlds are described by unboundedly-big numbers then of course you have exactly the same problem over worlds.

See Joe’s recent post On infinite ethics which prompted this post. I was especially responding to Part X which relied on the assumption that individual experiences can be arbitrarily good in order to argue that UDASSA-like schemes don’t really avoid the trouble with infinities. But I think they do avoid the distinctive trouble with infinitely-big universes, and that arbitrarily-good experiences are more deeply problematic in their own right.

[AlexMennen]

Replacing single utilities with time-indexed sequences of utilities doesn’t help for representing preferences. If you have to make a decision between two options, each of which will result in a different sequence of time-indexed utilities, you still need to decide which option is better overall, which means you’ll need a one-dimensional scale to compare these utility-sequences on. The VNM theorem tells you that, under certain fairly weak assumptions, a single real-valued utility is the appropriate measure to use for this.

[Slider]

Reals might not be “continuous enough” to do the job. A hard limit case.

Option A: 1 utility on day 1, 0 utility for the rest of days

Option B: 2 utility on day 1, 0 utility for the rest of days

Option C: 3 utility on day 1, 0 utility for the rest of days

Option D: 1 utility on day 1, 1 utiltity for the rest of days

Option E: 2 utility on day 1, 1 utility for the rest of days

Continuity means when $L<M<N$ then there should a p such that $pL+(1-p)N\sim M$

So there are values $p_{1}A+(1-p_1)C\sim B$ and $p_{2}A+(1-p_2)D\sim B$ and $p_{3}A+(1-p_3)E$. If $p_1$, $p_2$ adn $p_3$ are from the reals and different then they should be finite multiples of each other. So while one can do with one real to differentiate between A,B,C and D,E to me it seems the jump between the types of cases is not finite and the reals can’t provide that at the same time as keeping the resolution on differentiating betwen day 1 utilities.

With surreals the probabilities could be infinidesimal and the missing probabilities exist.

[AlexMennen]

If you have surreal-valued utilities, you can just round infinitesimals to 0 to get real-valued utilities, and then continuity can be satisfied with real-valued probabilities again. The resulting real-valued utility function is correct about your preferences whenever it assigns higher utility to one option than the other, and is deficient only in the case where it assigns the same utility to two different options that you value differently. But it is very unlikely for two arbitrary reals to be exactly the same, and even when this does happen, the difference is infinitesimally unimportant compared to other preferences, so this isn’t a big loss.

[Ruby]

Note: I’ve turned on two-axis voting for this post to see if it’s helpful for this kind of discussion.

[paulfchristiano]

Relevant comment from the sequences (I had this in mind when writing parts of the OP but didn’t remember who wrote it, and failed to recognize the link because it was about Newcomb’s problem):

Another example: It was argued by McGee that we must adopt bounded utility functions or be subject to “Dutch books” over infinite times. But: The utility function is not up for grabs. I love life without limit or upper bound: There is no finite amount of life lived N where I would prefer a 80.0001% probability of living N years to an 0.0001% chance of living a googolplex years and an 80% chance of living forever. This is a sufficient condition to imply that my utility function is unbounded. So I just have to figure out how to optimize for that morality. You can’t tell me, first, that above all I must conform to a particular ritual of cognition, and then that, if I conform to that ritual, I must change my morality to avoid being Dutch-booked. Toss out the losing ritual; don’t change the definition of winning. That’s like deciding to prefer $1000 to $1,000,000 so that Newcomb’s Problem doesn’t make your preferred ritual of cognition look bad.

I sympathize with Eliezer’s intuition here but think he’s overstating the case. (Setting aside the fact that the exact example isn’t correct, and that McGee’s particular dutch book—at least as described by Eliezer—seems very unpersuasive.)

I don’t know if Eliezer wants to give up on unbounded utilities, utility functions over arbitrary lotteries, or on weak dominance. One of them must go, and the others are supported by pretty good intuitions. Giving up on infinite lotteries is in some sense the mildest, but given that our epistemic states are infinite lotteries this is quite a bullet! Giving up on the epistemic possibility of very large universes (as discussed by several commenters) also seems conceivable but no more comforting than giving up on the utility function.

I don’t think Eliezer’s position here is altogether different from someone who has the strong intuitions that A > B > C > A, and strong intuitions about transitivity. Faced with the incoherence such a person could just say “well my preferences are my preferences, so be it,” but I feel confident they’d be making a mistake.

This is not to say that I know the answer, and I don’t think this case is as straightforward as intransitivity. But I don’t think it’s right to glibly dismiss this kind of impossibility argument because the utility function is not up for grabs. The utility function may not be up for grabs but intuitions about it are, and logical incoherence between intuitions is real evidence about those intuitions.

I also recommend this comment thread overall. (I actually think that “lifetime /​ size of universe” is a pretty good direction for bounded utility functions, and perhaps that’s Eliezer’s view?)

[MichaelStJules]

(Note: I’ve edited this comment a lot since first posting it, mostly small corrections, improving the definition of the order to be better behaved and adding more points.)

I think orders which satisfy (Symmetric) Unbounded Utilities but not (Weak) Dominance or Homogeneous Mixtures can be relatively nice, so we shouldn’t be too upset about giving up (Weak) Dominance and Homogenous Mixtures. Basically no nice order will be sensitive to the kinds of probability rearrangements you’ve done (which we can of course conclude from your results, if we want “nice” to include the other properties).

Here’s such an order, which also extends (is as strong as) expected utility and (I think) stochastic dominance for finite but possibly unbounded utilities and infinite expected utilities:

(EDIT: I think this order is pretty badly behaved based on how badly it violates Intermediate Mixtures, but this other one still seems promising.)

Given a utility function $U$ and lotteries $A$ and $B$, then $A<B$ if

$$\underset{(p,q)\to (0,1),0<p<q<1}{liminf}{\int }_p^q{Q}_{U\left(B\right)}\left(t\right)-Q_{U\left(A\right)}\left(t\right)dt>0,$$

where $Q_{U\left(A\right)}$ and $Q_{U\left(B\right)}$ are the quantile functions of $U\left(A\right)$ and $U\left(B\right)$, respectively. We can also say $A\le B$ when the above holds with $\ge 0$ instead of the strict $>0$.

Basically, take integrals out from the middle of the distributions (based on matching quantiles), and check if one dominates the other in the limit as you cover the whole probability space.

Some properties:

1. The order reduces to comparing expected utilities when the expected utilities of both lotteries are well-defined and they aren’t the same infinity (i.e. not both positive infinity or both negative infinity; in such cases, it can sometimes tell you one still dominates the other). This is because ${lim}_{(p,q)\to (0,1)}{\int }_p^q{Q}_X\left(t\right)dt={\int }_0^1{Q}_X\left(t\right)dt=E\left[X\right]$, whenever the expected value is well-defined.

2. The order is transitive (with bounded actual utilities, including when the expected utilities are infinite or undefined), because lim infs are superadditive, i.e. $liminff+g\ge liminff+liminfg$.

3. The order is not complete. It doesn’t rank $\frac{1}{2}{X}_{\infty }+\frac{1}{2}(1-X_{\infty })$ vs 0, where $1-X_{\infty }$ is just the lottery with the payoffs of $X_{\infty }$ with the negatives of the utilities and an extra utility of 1 added to each.

4. I’m not sure if the independence axiom holds, but I would guess so. It at least holds for lotteries with finite expected utility.

5. It doesn’t satisfy Dominance, Weak Dominance or Homogenous Mixtures because your counterexamples don’t work for it, since it’s not sensitive to probability rearrangements.

[davidad]

I think this order does satisfy Homogeneous Mixtures, but not Intermediate Mixtures. Homogeneous Mixtures is a theorem if you model lotteries as measures, because it’s asking that your preference ordering respect a straight-up equality of measures (which it must if it’s reflexive).

Intermediate Mixtures and Weak Dominance are asking that your preference ordering be willing to strictly order mixtures if it would strictly order their components in a certain way, and the ordering you’ve proposed preserves sanity by sometimes refusing to rank pathological mixtures.

[MichaelStJules]

Hmm, I do think Intermediate Mixtures is violated, and in a very bad way: it can flip the order.

Consider

1. $A=0$, constant

2. $B={\sum }_{i=1}^{\infty }\frac{1}{2^i}{\delta }_{3^i}=3^i$ with probability $2^i$ for each $i=1,2,\dots$.

Note that $B>A$.

Let’s check if $B>\frac{1}{2}A+\frac{1}{2}B$. For a given $q$ sufficiently close to 1 (away from ¹⁄₂, at least), the integral of the integral of the quantile for $\frac{1}{2}A+\frac{1}{2}B$ will be much further into the series terms of $B$ than the integral of the quantile $B$, because $\frac{1}{2}A+\frac{1}{2}B$ has to cram the probabilities of $B$ into half as much space. Because the expected value of the terms grow exponentially, halving the probabilities is outweighed by summing the series at a faster rate (with respect to the probabilities). In other words, the quantile integral of $\frac{1}{2}A+\frac{1}{2}B$diverges to infinity faster than $B$’s.

So, I think it’s actually the case that $B<\frac{1}{2}A+\frac{1}{2}B$, which seems very bad, bad enough to reject this approach.

(I think the order I defined here will be better behaved.)

[MichaelStJules]

Which equality (not preference equivalence) of measures are you talking about for Homogenous Mixtures?

The order doesn’t satisfy Homogenous Mixtures, but maybe it also doesn’t satisfy Intermediate Mixtures. For Homogenous Mixtures, using lotteries over actual utilities, i.e. the payoff/​outcome is its utility, for $i=0,1,2,\dots$,

1. $X=0$

2. $A_i=\frac{1}{2}{\delta }_{2^i}+\frac{1}{2}{\delta }_{-2^i}=2^i\text{or}-2^i,$ each with probability ¹⁄₂. This is equivalent to $X=0$ since we’re ranking based on expected utility when both lotteries are bounded.

3. $p_i=\frac{1}{2^{i+1}}$

It doesn’t order ${\sum }_{i=1}^{\infty }{p}_i{A}_i$ vs $X$, because fixing $q$ and taking $p\to 0$, the integral for ${\sum }_i{p}_i{A}_i\ge 0$ diverges to $-\infty$, and the integral for $0\ge {\sum }_i{p}_i{A}_i$ diverges to $-\infty$. Note that

${\sum }_{i=1}^{\infty }{p}_i{A}_i=\frac{1}{2}{A}_{+}+\frac{1}{2}{A}_{-}$, where

$A_{+}={\sum }_{i=0}^{\infty }\frac{1}{2^{i+1}}{\delta }_{2^i},A_{-}={\sum }_{i=0}^{\infty }\frac{1}{2^{i+1}}{\delta }_{-2^i}$,

so a mixture of two lotteries, one diverging to $+\infty$, and the other diverging to $-\infty$.

On the other hand, if you require $q=1-p$, then the integral will actually be 0 for each $p$ for this particular lottery, since the lottery is symmetric around $p=\frac{1}{2}$, so you do get equivalence. I suspect we can come up with another counterexample by messing around with how fast each tail is approached and get the liminf of the integral to be positive, and so rank ${\sum }_i{p}_i{A}_i>0$. Maybe instead defining $A_i$ this way would work:

$A_i=(1-\frac{1}{1+2^i}){\delta }_{2^i}+\frac{1}{1+2^i}{\delta }_{-4^i}$.

The idea is that the negative terms get much less far for the same $p$, because far more of the weight is in much lower probability events. The integral for ${\sum }_i{p}_i{A}_i\ge 0$ is roughly counting the number of positive terms whose probability is above $cp$ and subtracting the number of negative terms whose probability is above $cp$, for some constant $c$ (I can’t be bothered to figure out exactly which $c$). This should go to $+\infty$ as $p\to 0$.

I think you can make things worse, too, again with $q=1-p$. You can choose $A_i<0$ for each $i$, but have ${\sum }_i{p}_i{A}_i>0$ by replacing the $-4^i$ with $-4^i-1$. I think we can even get the gap between $A_i$ and 0 to diverge as $i\to \infty$, with something like $-4^i-i(1+2^i)$ or even $-4^i-3^i$ instead of $-4^i$. If we allow $p$ and $q$ to vary independently again, ${\sum }_i{p}_i{A}_i$ and 0 are just incomparable, which seems okay.

Unfortunately, the quantile function is a pain to work with, so checking whether Intermediate Mixtures holds seems tricky.

[MichaelStJules]

Here are some ways to get more strict inequalities (less incomparability or equivalence):

1. Require $q=1-p$ to handle some more cases with both positive and negative expected infinities, but I’m not sure that the results would always be intuitive. There might be other relationships between $p$ and $q$ that that depend on the particular lotteries that work better. You could test the lim infs under multiple relationships, $q=1-f\left(p\right)$ for different $f$ from a specific set.

2. Replace the strict inequality condition with

$$\underset{(p,q)\to (0,1),0<p<q<1}{lim}\text{sgn}({\int }_p^q{Q}_{U\left(B\right)}\left(t\right)-Q_{U\left(A\right)}\left(t\right)dt)=1.$$

Equivalently, there are $p_0,q_0,0<p_0<q_0<1$ such that ${\int }_p^q{Q}_{U\left(B\right)}\left(t\right)-Q_{U\left(A\right)}\left(t\right)>0$ for all $p,q,0<p<p_0<q_0<q<1$.

$A<B$ would mean that the integral for $A$ never catches up with that for $B$ in the limit.

[MichaelStJules]

p.37-38 in Goodsell, 2023 gives a better proposal, which is to clip/​truncate the utilities into the range $[-t,t]$ and compare the expected clipped utilities in the limit as $t\to \infty$. This will still suffer from St Petersburg lottery problems, though.

[TekhneMakre]

Maybe some of the problem is coming from trying to extend dominance to infinite combinations of lotteries. If we’re saying that the utility function is the thing that witnesses some coherence in the choices we make between lotteries, maybe it makes sense to ask for choices between finite combinations of lotteries but not infinite ones? Any choice we actually make, we end up making by doing some “finitary” sort of computation (not sure what this really means, if anything), and perhaps in particular it’s always understandable as a choice between finite lotteries. Like, if we choose between two lotteries, at that time we only had a finite number of concepts in mind, which implies a finite set of possible descriptions of lotteries, so that all lotteries we can distinguish are finite combinations of those finitely many lotteries.

Can the paradox be extended to someone who rejects Weak Dominance but accepts finitary dominance? E.g. by offering an infinite sequence of choices between finite combinations of lotteries, or something?

[paulfchristiano]

I think you avoid any contradiction if you reject Weak Dominance but accept a finite version of Dominance. For example, in that case you can simply declare all lotteries with infinite support to be incomparable to each other or to any finite lottery.

If you furthermore require your preferences to be complete, even when asking about infinite lotteries, such that either $A>B$ or $A>B$ or $A=B$, then I suspect you are back in trouble.

But if you just restrict preferences to finite lotteries you are fine and can compare them with expected value.

[TekhneMakre]

Yeah, maybe just truncating off finitely many summands in an infinite lottery induces constraints that force your examples to have infinite value? Maybe you can have complete hyperreal-valued preferences and finite dominance...?

[Richard_Ngo]

Another common way out is to assume that any two “infinitely good” outcomes are incomparable, and therefore to reject Dominance.^[3] This results in being indifferent to receiving $1 in every world (if the expectation is already infinite), or doubling the probability of all good worlds, which seems pretty unsatisfying.

Why are these unsatisfying? Intuitively, if I already have infinite money (in expectation) then why should I care about getting to infinite + 1 or infinite x 2 money?

The point you mention about all decisions having infinite utility in expectation does seem worrying though—do you have an accessible intuition for why this is the case?

[paulfchristiano]

To take it further, suppose that with 1% probability you are able to play a St. Petersburg game, and in the other 99% of worlds there is a billion years of torture. Then the story is that you don’t care about whether the probabilities are 1% and 99%, or 99% and 1%. Whether or not you find that unsatisfying is a personal call, but I find it extremely bad.

(But this proof doesn’t show that’s an inevitable consequence of Unbounded Utilities, it just shows that violating Dominance is an inevitable conclusion. So you might well think that this torture case is pretty unsatisfying but you can take or leave Dominance itself. I think that’s not crazy, but I think you’d be able to run a similar argument to get to any particular unsatisfying Dominance-violation.)

(I personally find violating Weak Dominance much more surprising, and that’s the point where I’m saying that you should just give up on talking about probabilistic mixtures. Though that may be too drastic. I’m phrasing this whole post in terms of dominance principles because I want to make the point that unbounded utilities basically force you to abandon very basic parts of your decision-theoretic machinery so you shouldn’t go on as if you have unbounded utilities but an otherwise normal decision theory.)

The point you mention about all decisions having infinite utility in expectation does seem worrying though—do you have an accessible intuition for why this is the case?

Basically just Pascal’s mugging. Under universal /​ non-dogmatic distributions, there is some probability on “Someone controls the universe, specifically searches for a series of outcomes with really large utility, and then runs the St. Petersburg game.”

(Of course for aggregative utilitarians you don’t even need to go there, any not-insane probability distribution over the size of the reachable universe is just obviously going to have infinite expectation.)

[Signer]

If infinitely valuable outcome is possible at all (has non-zero probability) after decision, then multiplying infinite utility by any non-zero probability you always get infinite expected utility.

[Slider]

if the system allows infinidesimals this need not be the case

[Matthew Barnett]

This argument is extremely similar to Beckstead and Thomas’ argument against Recklessness in A paradox for tiny probabilities and enormous values.

If I understand correctly, this argument also appeared in Eliezer Yudkowsky’s post “The Lifespan Dilemma”, which itself credits one of Wei Dai’s comment here. The argument given in The Lifespan Dilemma is essentially identical to the argument in Beckstead and Thomas’ paper.

[paulfchristiano]

I think Eliezer and Wei Dai’s comments (and the early part of Beckstead and Thomas) are just direct intuitive arguments against Recklessness.

This post (and the later part of Beckstead and Thomas) argue that Recklessness is not merely intuitively unappealing, but that it requires violating pretty weak dominance principles. You have to believe that there is a set of lotteries $A_i$ each individually better than $X$, whose mixture is not at least as good as $X$.

Someone who already bought the intuitive argument against Recklessness doesn’t need to read these posts; they are for someone who already bit the bullet on the lifespan dilemma and wants more bullets.

[James Payor]

I spent some time trying to fight these results, but have failed!

Specifically, my intuition said we should just be able to look at the flattened distributions-over-outcomes. Then obviously the rewriting makes no difference, and the question is whether we can still provide a reasonable decision criterion when the probabilities and utilities don’t line up exactly. To do so we need some defined order or limiting process for comparing these infinite lotteries.

My thought was to use something like “choose the lottery whose samples look better”. For instance, examine ${lim}_{n\to \infty }P[{\sum }_{0\le i<n}{A}_i-B_i\le 0]$, where $A_i$ and $B_i$ are samples from two lotteries $A$ and $B$. This should prioritize more extreme low-probability events only as $n$ grows larger. This works for comparing things against a standard St Petersburg lottery.

But the problem I’m facing is the lottery $\frac{1}{2}1-\frac{1}{4}4+\frac{1}{8}16-\frac{1}{16}64+\cdots$. If you try to compare it to any finite payoff like 500, when the samples get big enough it will wildly swing one way or the other, so we can’t tell with my method which we “prefer”. (In real life I’d take the $500, fwiw.)

My intuition complains that this lottery is rather unfair, but nonetheless it is sinking my brilliant idea! So my current guess is that I can handle things when probability-times-utility is eventually bounded. But I don’t know how to cope with utilities growing faster than probabilities shrink.

[Sonata Green]

One interesting case where this theorem doesn’t apply would be if there are only finitely many possible outcomes. This is physically plausible: consider multiplying the maximum data density¹ by the spacetime hypervolume of your future light cone from now until the heat death of the universe.

¹ <https://​​physics.stackexchange.com/​​questions/​​2281/​​maximum-theoretical-data-density>

[NormanPerlmutter]

I just quickly browsed this post. Based on the overall topic, you might also be interested in these inconsistency results in infinitary utiliatarianism written by my PhD advisor (a set theorist) and his wife (a philosopher).

http://​​jdh.hamkins.org/​​infinitary-utilitarianism/​​

[chasmani]

Your proofs all rely on lotteries over infinite numbers of outcomes. Is that necessary? Maybe a restriction to finite lotteries avoids the paradox.

[JBlack]

Thank you for making explicit the idea that the problems with “unbounded utilities” don’t even require the existence of a utility function, just a very weak assumption about preference ordering with respect to probability mixtures and the existence of “arbitrarily strong” outcomes.

[Sniffnoy]

I should note that this is more or less the same thing that Alex Mennen and I have been pointing out for quite some time, even if the exact framework is a little different. You can’t both have unbounded utilities, and insist that expected utility works for infinite gambles.

IMO the correct thing to abandon is unbounded utilities, but whatever assumption you choose to abandon, the basic argument is an old one due to Fisher, and I’ve discussed it in previous posts! (Even if the framework is a little different here, this seems essentially similar.)

I’m glad to see other people are finally taking the issue seriously, at least...

[paulfchristiano]

I agree that “unbounded utilities” don’t refer to anything at all in the usual sense of “utility function” and that this observation is basically as old as VNM itself.

I usually cite de Blanc 2007 to point out that unbounded utilities are just totally busted for non-dogmatic priors (but this is also a formalization of a much older argument about “contagion”).

The point of these posts was to observe that this isn’t just an artifact of utility functions, and that changing the formalism doesn’t help you get around the problems. So this isn’t really an argument against utility functions, it’s a much more direct argument against a certain kind of preferences. There just don’t exist any transitive preferences with unbounded-utility-like-behavior and weak outcome-lottery dominance.

[Sniffnoy]

Oh, that’s a good citation, thanks. I’ve used that rough argument in the past, knowing I’d copied it from someone, but I had no recollection of what specifically or that it had been made more formal. Now I know!

My comment above was largely just intended as “how come nobody listens when I say it?” grumbling. :P

[habryka]

Promoted to curated: I’ve been thinking about unbounded utilities for a while, and while I agree with a bunch of the top commenters on the actual relevant element being the integrateability of utilities, and that this can save some unbounded utility assignments (in a way that feels relevant to my current beliefs on the topic).

I do think nevertheless that this post is the best distillation of a bunch of the impossibility arguments I’ve seen floating around for the last decade, and I think it is an actually important question when trying to decide how to relate to the future. So curating it seems appropriate.

[MichaelStJules]

The examples and results in your post are very interesting and surprising. Thanks for writing this.

I’m inclined to reject the dominance axioms you’ve assumed, at least for mixtures of infinitely many lotteries. I think stochastic dominance is a more fundamental axiom, avoids inconsistency and doesn’t give any obviously wrong answers on finite payoff lotteries (even mixtures of infinitely many lotteries or outcomes with intuitively infinite expected value, including St. Petersburg and Pasadena). See Christian Tarsney’s “Exceeding Expectations: Stochastic Dominance as a General Decision Theory” (skip sections 5-7, which are about applying it to models with background uncertainty; the definitions and initial motivation are in section 3, quoted in my reply, and various choice situations are considered in section 9). I think the independence axiom and your dominance axioms restricted to mixtures of finitely many lotteries (almost?) follow from versions of stochastic dominance extended to sequences of lotteries and multiple decisions, based on Dutch book/​money pump arguments. See Johan E. Gustafsson’s “The Sequential Dominance Argument for the Independence Axiom of Expected Utility Theory”.

Without good enough ways to distinguish different definite infinite payoffs (i.e. first solving infinite ethics, although not all solutions to infinite ethics would necessarily fit well with stochastic dominance), stochastic dominance can give unintuitive results when comparing lotteries with intuitively different infinite payoffs. It could also allow for incomparability, but a kind of incomparability that doesn’t necessarily break everything (well, not (much) more than infinite ethics already breaks things), since it just means multiple options can be permissible within a given set of options, even if they can’t be treated as if they were equivalent (when mixing with other lotteries, considering background noise, etc.) across all sets of options.

[paulfchristiano]

If we define $A<B$ whenever $B$ stochastically dominates $A$, then I think that you have Dominance (since mixtures preserve stochastic dominance) but not Unbounded Utilities (since it’s impossible for a smaller chance of a good outcome to dominate a higher chance of a less-good outcome), right?

[MichaelStJules]

If you’re defining the order purely based on stochastic dominance (and no stronger), then ya, I think you’ll have Dominance but not Unbounded Utilities for the reasons you give.

However, I think stochastic dominance is consistent with Unbounded Utilities in general and the expected utility order when choosing between bounded lotteries, since the order based on expected utilities is well-defined and stronger than the stochastic dominance one over bounded lotteries. That is, if $A$ strictly (or weakly) stochastically dominates $B$, and both are bounded lotteries, then $A$ has a higher expected utility than $B$ (or, at least as high, respectively). So, you could use an order that’s at least as strong as using expected utilities, when they’re well-defined, and also generally at least as strong as stochastic dominance, but that doesn’t imply Dominance for mixtures of infinitely many lotteries. Your specific example with $X_{\infty }$ would prove that Dominance does not hold.

Also, if you’re extending expected utility anyway, you’d probably want to go with something stronger than stochastic dominance, something that also implies sequential dominance or some kind of independence.

[MichaelStJules]

EDIT: p.37-38 in Goodsell, 2023 gives a better proposal, which is to clip/​truncate the utilities into the range $[-t,t]$ and compare the expected clipped utilities in the limit as $t\to \infty$. This will still suffer from St Petersburg lottery problems, though.

Here’s an order that’s as strong as both expected utility and stochastic dominance, and overall seems promising to me:

tl;dr: For lotteries with finite utility payoffs (but possibly unbounded utility payoffs and infinite expected utility), we can take expectations through any subset with finite and well-defined expected utility, and then compare the resulting lotteries with stochastic dominance. We just need to find any pair of well-behaved “expected utility collapses” for which one lottery stochastically dominates the other. Allowing expected utility collapses over the infinite expected utilities can lead to $A<A$, so I rule that out.

In practice, you might just take one expectation over everything but the top X% and bottom Y% of each lottery, and compare those lotteries with stochastic dominance, for different values of X and Y. This allows you to focus on the tails of heavy-tailed distributions.

For a lottery $X$, a utility function $U$, and a countable (possibly finite and possibly empty) set of mutually exclusive non-empty measurable subsets of the measure space, $P=\{Q_1,Q_2,\dots ,Q_n\}$ (or basically a set of binary random variables whose sum is at most 1) and letting $P^C=({\cup }_{Q\in P}Q{)}^C$ be the complement of their union (so, for their indicator binary random variables, $1_{P^C}=1-{\sum }_{Q\in P}{1}_Q$), the expected utility collapse of $X$ over $P$ is:

$X_P=E\left[U\right(X\left)|Q\right]$ if $Q$, for $Q\in P$, and $X_P=X|P^C$, otherwise.

Or, in lottery notation, letting $L\left(c\right)$ be the constant lottery with constant value $c$,

$X_P=P\left(P^C\right)X|P^C+{\sum }_{Q\in P}P\left(Q\right)L\left(E\right[U\left(X\right)|Q\left]\right)$.

In other words, we replace probability subsets of $X$ with its expected utility over those subsets.

If furthermore, $E\left[U\right(X\left)|Q\right]$ is well-defined and finite for each $Q\in P$, we call the expected utility collapse well-behaved.

Then, we define the order over lotteries as follows:

$A<B$ if there exists well-behaved expected utility collapses $A_{P_1}$ and $B_{P_2}$ of $A$ and $B$ respectively such that $B_{P_2}$ strictly stochastically dominates $A_{P_1}$.

If you allow infinite actual utilities (including possibly infinities of different magnitudes), you could add a disjunctive or overriding condition to handle comparisons with those.

[MichaelStJules]

From section 3 of Tarsney’s paper:

Option O first-order stochastically dominates option P if and only if

1. For any payoff x, the probability that O yields a payoff at least as good as x is equal to or greater than the probability that P yields a payoff at least as good as x, and

2. For some payoff x, the probability that O yields a payoff at least as good as x is strictly greater than the probability that P yields a payoff at least as good as x.

(...)

Stochastic dominance is a generalization of the familiar statewise dominance relation that holds between O and P whenever O yields at least as good a payoff as P in every possible state, and a strictly better payoff in some state. To illustrate: Suppose that I am going to flip a fair coin, and I offer you a choice of two tickets. The Heads ticket will pay $1 for heads and nothing for tails, while the Tails ticket will pay $2 for tails and nothing for heads. The Tails ticket does not statewise dominate the Heads ticket because, if the coin lands Heads, the Heads ticket yields a better payoff. But the Tails ticket does stochastically dominate the Heads ticket. There are three possible payoffs: winning $0, winning $1, and winning $2. The two tickets offer the same probability of a payoff at least as good as $0, namely 1. And they offer the same probability of an payoff at least as good as $1, namely 0.5. But the Tails ticket offers a greater probability of a payoff at least as good as $2, namely 0.5 rather than 0. Stochastic dominance is generally seen as giving a necessary condition for rational choice:

Stochastic Dominance Requirement (SDR) An option O is rationally permissible in situation S only if it is not stochastically dominated by any other option in S.

This principle is on a strong a priori footing. Various formal arguments can be made in its favor. For instance, if O stochastically dominates P, then O can be made to statewise dominate P by an appropriate permutation of equiprobable states in a sufficiently finegrained partition of the state space (Easwaran, 2014; Bader, 2018).

[deleted]

[TLW]

This doesn’t hold if you restrict to utility functions that asymptote to a finite value, correct?

[LGS]

Very nice, thanks. I agree with others that if one were intent on keeping unbounded utilities, it seems simplest to give up on probability distributions that have infinite support (it seems one can avoid all these paradoxes by restricting oneself to finite-support distributions only). I guess this is similar to what you mean by “abandoning probability itself”, but do note that you can keep probability so long as all the supports are finite.