I don’t see why it would make more sense to speak of the probability that you are the nth observer given some theory than the probability that you are a rock or that you are a nonsentient AI. If you object that anthropic reasoning is about consciousness, you’d have to bite the bullet and say that doomsday argument-like reasoning is invalid for nonsentient computer programs.
Also, if you’re interested in anthropics, I recommend this paper by the Future of Humanity Institute’s Stuart Armstrong.
Clearly, you can define whatever theory you like. But to be testable, the theory has to make some predictions about our observations. For this, it is not sufficient for a theory to describe in “third party” terms what events objectively happen in the world. There also has to be a “first party” (or “indexical”) component to answer the question: OK then, but what should we expect to observe?
As an analogy, you could treat the third party component as an objectively accurate map, and the first party component as a “You are here” marker placed on the map. The map is going to be pretty useless without the marker.
Similarly, if a theory of the world doesn’t have a “first party” component, then in general it is not possible to extract predictions about what we should be observing. This is especially true if the “third party” component describes a very large of infinite universe (or multiverse). Further, if the “first party” component predicts that we should be rocks or nonsentenient AIs, well then it seems that we can falsify that component straight away. This would be similar to a “you are here” marker out in the ocean somewhere, when we know we are on dry land.
Finally, just to advise, my post was not a general defence of the Doomsday Argument, but rather a discussion on whether the SIA “move” works to defuse its conclusions. It’s quite noticeable in the DA literature that lots of people are really sure that there is something wrong with the DA, and they give all sorts of different diagnoses as to where the flaw lies, but then most of these objections either turn out not to work, or to be objections against Bayesian inference full stop. There has been—as far as I can tell—a general view that the SIA does work as a way of defusing the DA, and so it is a real objection to the DA, if you accept the SIA. The discussion then diverts into whether it is legitimate to accept the SIA.
But I disagree with that view. My analysis is effectively this: “Fine, I will grant you the SIA for the sake of argument. So you now have an infinite world. But then—oh dear—on close inspection, it looks like you haven’t defused the DA after all. In fact you just made it even stronger.”
This comment made me think about anthropics, but I never got back to you about my conclusions. What I decided is that the first party component looks really important, but it is irrelevant to decision making (except insofar as our utility functions value the existence of conscious beings). For example, if one world has 100 of me and the other has only one, I might want to precommit to some strategy based on just a Solomonoff prior or similar, with no anthropic considerations. If I do that, I wouldn’t want to change my mind based on the additional knowledge that I exist. That was rather jargonny, so see some introductions to UDT or an illustrative though experiment.
I therefore disagree with your conclusion about the DA. If Omega tells me that he is going to create either world 3.1 or world 3.2, I would be surprised in some sense to find myself as one of the first beings in world 3.2, but finding out that I am one of the first 200 billion observers does not give me any reason to violate a rationally chosen precommitment. This seems odd, but the decision-theoretic considerations necessitated by the various thought experiments associated with UDT just seem to be a different sort of thing than first person experience. (I’m referring to the thought experiments specifically mentioned in the above links, in case that isn’t clear. Also, this isn’t in those links, but it might be illustrative to consider a nonsentient optimization process. Such a thing clearly does not have first-person experience and isn’t in our reference class, whatever that means, but it can still apply decision theory to achieve its goals.)
I’m not sure I get this. I think I’ve grasped the high-level point about UDT (that the epistemic probabilities strictly never update). So that if a UDT agent has a Solomonoff prior, they always use that prior to make decisions, regardless of evidence observed.
However, UDT agents have still got to bet in some cases, and they still observe evidence which can influence their bets. Suppose that each UDT agent is offered a betting slip which pays out 1 utile if the world is consistent with H3.1, and nothing otherwise. Suppose an agent has observed or remembers evidence E. How much in utiles should the agent be prepared to pay for that slip? If she pays x (< 1) utiles, then doesn’t that define a form of subjective probability P[H3.1|E] = x? And doesn’t that x vary with the evidence?
Let’s try to step it through. Suppose in the Solomonoff prior that H3.1 has a probability p31 and H3.2 has a probability p32. Suppose also that the probability of a world containing self-aware agents who have discovered UDT is pu and the probability of an infinite world with such agents is pui.
Suppose now that an agent is aware of its own existence, and has reasoned its way to UDT but doesn’t yet know anything much else about the world; it certainly doesn’t know yet how many observers there have been yet in its own civilization. Let’s call this evidence E0.
Should the agent currently pay p31 for the betting slip, or pay something different as her value of P[H3.1 | E0]?
If something different, then what? (A first guess is that it is p31/pu; an alternative guess is p31/pui if the agent is effectively applying SIA). Also, at this point, how much should the agent pay for a bet that pays off if the world is infinite: would that be pui/pu or close to 1?
Now suppose the agent learns that she is the 100 billionth observer in her civilization, creating evidence E1. How much should the agent now pay for the betting slip as the value of P[H3.1| E1]? How much should the agent pay for the infinite bet?
Finally, do the answers depend at all on the form of the utility function, and on whether correct bets by other UDT agents add to that utility function? From my understanding of Stuart Armstrong’s paper, the form of the utility function does matter in general, but does UDT make any difference here? (If the utility depends on other agents’ correct bets, then we need to be especially careful in the case of worlds with infinitely many agents, since we are summing over infinitely many wins or losses).
That analysis uses standard probability theory and decision theory, but that doesn’t work in this sort of situation.
Compare this to Psy-Kosh’s non-anthropic problem. Before you are told whether you are a decider, you can see, in the normal way, that it is better to follow the strategy of choosing “nay” rather than “yea” no matter what. If you condition on finding out that you are a decider the same way that you would condition on any piece of evidence, it appears that it would be better to choose “yea”, but we can see that someone following the strategy of updating like that will get less utility, in expectation, than someone who follows the strategy of choosing “nay” in response to any evidence. Therefore, decision theory as usual leads to suboptimal outcomes in situations like this.
In your situation, we can follow the same principles to see that standard decision can fail there too, depending on your utility function. From a UDT perspective, you are choosing between the strategies “If you have a low birth rank, be willing to pay almost a full util, if necessary, for the betting slip.” and “Only buy the slip if it is reasonably priced, ie. costs < p31 utils, no matter what you observe.” and you should weigh the resulting utility differences using the prior (p31, p32) for hypotheses H3.1 and H3.2 that you assign before you observe your birth rank. A utilitarian would know, before observing their actual birth rank, that they would obtain the highest utility by following the second strategy, so updating as you suggest would give a different answer than the one that looks best in advance, in the same way that updating in the usual way in Psy-Kosh’s problem produces a suboptimal outcome. Stuart goes into a bit more detail on this, discussing more types of utility functions and doing more of the math explicitly.
You seem to be arguing here that a UDT agent should NEVER update their betting probabilities (i.e. never change the amount paid for a betting slip) regardless of evidence. This seems plain wrong to me in general e.g. imagine I’m offered a lottery ticket for $1 a few minutes after the draw and I already saw the numbers drawn; if they match the numbers on the ticket, this is a great deal and I should take it. If UDT really says that I shouldn’t, then I’m sticking with CDT!
So I don’t think that is what you are arguing (i.e. don’t update betting probabilities in general); but you are arguing not to update the betting probability in this case. Correct?
Here is another example to consider, based on other types of evidence. Hypothesis B1 is that the universe is infinite, homogeneous, isotropic and has a background radiation of 1K. Hypothesis B3 is the same, except with background radiation of 3K. These have Solomonoff prior probabilities of pb1 and pb3 respectively. A UDT agent measures the background radiation and it comes out at 3K. Does a UDT agent still pay pb1 for the betting slip which pays out if B1 is correct? (Remember that there are infinitely many UDT agents in both B1 and B3 worlds, and infinitely many agents in each case will have just made a 3K measurement, because of measurement errors in the 1K world. The agent still doesn’t know for sure which sort of world she’s in.)
I presume you will want the agent to change betting probabilities in the background radiation case (because if she doesn’t, she’s about to lose against a CDT agent who does e.g. when betting on subsequent measurements)? And if it’s right to change in that case, why isn’t it right to change in the H31/H32 case?
Yes, I do think you should update in that case. The one-sentence version of UDT (UDT1.1 more precisely) is that you precommit to the best possible strategy and then follow that strategy. The strategy is allowed to recommend updating. In fact, the agent in my previous example explicitly considered the strategy “If you have a low birth rank, be willing to pay almost a full util, if necessary, for the betting slip.” which tells the agent to make different bets depending on what they observe. This strategy was not chosen because the agent thought that it would result in a lower expected utility, but in other situations, such as the example you just presented, the optimal strategy does entail taking different actions in response to evidence. In many thought experiments, UDT adds up to exactly what you would expect.
Actually, I’m not sure whether the strategy that I selected is optimal. It would be optimal if a single world of either the type mentioned in H3.1 or the type from H3.2 existed, but we can’t sum that over an infinite number of worlds since the sum diverges. We could switch to a bounded utiliy function, but that’s a different issue entirely.
Actually, I’m not sure whether the strategy that I selected is optimal.
I’m pretty sure that it isn’t optimal, and for a much simpler reason than having infinitely many worlds. The strategy of
“Only buy the slip if it is reasonably priced, ie. costs < p31 utils, no matter what you observe” leads to a Dutch Book.
This takes a bit of explaining, so I’ll try to simplify.
First let’s suppose that the two hypotheses are the only candidates, and that in prior probability they have equal probability 1⁄2.
H3.1. Across all of space time, there are infinitely many civilizations of observers, but the mean number of observers per civilization (taking a suitable limit construction to define the mean) is 200 billion observers.
H3.2. Across all of space time, there are infinitely many civilizations of observers, but the mean number of observers per civilization (taking the same limit construction) is 200 billion trillion observers.
We’ll also suppose that both H3.1 and H3.2 imply the existence of self-aware observers who have reasoned their way to UDT (call such a being a “UDT agent”), and slightly simplify the evidence sets E0 and E1:
E0. A UDT agent is aware of its own existence, but doesn’t yet know anything much else about the world; it certainly doesn’t know yet how many observers there have been yet in its own civilization.
(If you’re reading Stuart Armstrong’s paper, this corresponds to the “ignorant rational baby stage”).
E1. A UDT agent discovers that it is among the first quadrillion (thousand trillion) observers of its civilization.
Again, we define P[X|Ei] as the utility that the agent will pay for a betting slip which pays off 1 utile in the event that hypothesis X is true. You are proposing the following (no updating):
Now, what does the agent assign to P[E1 | E0]? Imagine that the agent is facing a bet as follows. “Omega is about to tell you how many observers there have been before you in your civilization. This betting slip will pay 1 utile if that number is less than 1 quadrillion.”
It seems clear that P[H3.1 & E1 | E0] is very close to P[H3.1 | E0]. If H3.1 is true, then essentially all observers will learn that their observer-rank is less than one quadrillion (forget about the tiny tail probability for now).
It also seems clear that P[H3.2 & E1| E0] is very close to zero, since if H3.2 is true, only a miniscule fraction of observers will learn that their observer-rank is less than one quadrillion (again forget the tiny tail probability).
So to good approximation, we have betting probabilities P[E1 & H3.1 | E0] = P[E1 | E0] = P[H3.1 | E0] = 1⁄2 and
P[~E1 | E0] = 1⁄2. Thus the agent should pay 1⁄2 for a betting slip which pays out 1 utile in the event E1 & H3.1. The agent should also pay 1⁄4 for a betting slip which pays out 1⁄2 utile in the event ~E1.
Now, suppose the agent learns E1. According to your proposal, the agent still has P[H3.1 | E1] = 1⁄2, so the agent should now be prepared to sell the betting slip for E1 & H3.1 for the same price that she paid for it i.e. she sells it again for 1⁄2 a utile.
Oops: the agent is now guaranteed to lose 1⁄4 utile in all circumstances, regardless of whether or not she learns E1 or ~E1. If she learns ~E1, then she pays 3⁄4 for her two bets and wins the ~E1 bet, for a net loss of 1⁄4. If she learns E1 then she loses the bet on ~E1 and her bet on H3.1 & E1 is cancelled out since she has bought and sold the slip at the same price.
Incidentally, Stuart Armstrong discusses this issue in connection with the “Adam and Eve” problem, though he doesn’t give an explicit example of the Dutch book (I had to construct one). The resolution Stuart proposes is that an agent in the E0 (“ignorant rational baby”) stage should precommit not to sell the betting slip again if she learns E1 (or, strictly, not to sell it again unless the sale price is very close to 1 utile). Since we are discussing UDT agents, no such precommitment is needed; the agent will do whatever she should have precommited to do.
In practice this means that on learning E1, the agent follows the commitment and sets her betting probability for H3.1 very close to 1. This is, of course, a Doomsday shift.
You’re still, y’know, updating. Consider each of these bets from the updateless perspective, as strategies to be willing to accept such bets.
The first bet is to “pay 1⁄2 for a betting slip which pays out 1 utile in the event E1 & H3.1”. Adopting the strategy of accepting this kind of bet would result in 1⁄2 util for an infinite number of beings and −1/2 util for an infinite number of beings if H3.1 is true and would result in −1/2 util for an infinite number of beings if H3.2 is true.
If we could aggregate the utilities here, we could just take an expectation by weighing them according to the prior (equally in this case) and accept the bet iff the result was positive. This would give consistent, un-Dutch-bookable, results; since expectations sum, the sum of three bets with nonnegative expectations must itself have a nonnegative expectation. Unfortunately, we can’t do this since, unless you come up with some weird aggregation method other than total or average for the utility function (though my language above basically presumed totalling), the utility is a divergent series and reordering divergent series changes their sums. There is no correct ordering of the people in this scenario, so there is no correct value of the expected utility.
Moving on to the second bet, “pay 1⁄4 for a betting slip which pays out 1⁄2 utile in the event ~E1”, we see that the strategy of accepting this gives +1/2 infinitely many times and −1/2 infinitely many times if H3.1 is true and it gives −1/2 infinitely many times if H3.2 is true. Again, we can’t do the sums.
Finally the third bet, rephrased as a component of a strategy, would be to sell the betting slip from the first bet back for 1⁄2 util again if E1 is observed. Presumably, this opportunity is not offered if ¬E1, so there is no need for the agent to decide what to do in this case. This gives −1/2 infinitely many times if H3.1 and +1/2 infinitely many times if H3.2. The value of 1⁄2 −1/2 ∞ + 1⁄21⁄2 ∞ is, of course, indeterminate, so we can again neither recommend accepting or declining this bet without a better treatment of infinities.
I’m being careful to define the expressions P[X|Ei] as the amount paid for a betting slip on X in an evidential state Ei. This is NOT the same as the agent’s credence in hypothesis X. I agree with you that credences don’t update in UDT (that’s sort of the point). However, I’m arguing that betting payments must change (“update” if you like) between the two evidential states, or else the agent will get Dutch booked.
You describe your strategy as having an infinite gain or loss in each case, so you don’t know whether it is correct (indeed you don’t know which strategy is correct for the same reason). However, earlier up in the thread I already explained that this problem will arise if an agent’s utility depends on bets won or lost by other agents. If instead, each agent has a private utility function (and there is no addition/subtraction for other agents’ bets; only for her own) then this “adding infinities” problem doesn’t arise. Under your proposed strategy (same betting payments in E0 and E1), each individual agent gets Dutch-booked and makes a guaranteed loss of 1⁄4 utile so it can’t be the optimal strategy.
What is optimal then? In the private utility case (utility is a function only of the agent’s own bets), the optimal strategy looks to be to commit to SSA betting odds (which in the simplified example means an evens bet in the state E0, and a Doomsday betting shift in the state E1).
If the agent’s utility function is an average over all bets actually made in a world (average utilitarianism) then provided we take a sensible way of defining the average, such as take the mean (betting gain—betting loss) over N Hubble volumes, then take the limit as N goes to infinity, the optimal strategy is again SSA betting odds.
If the agent’s utility function is a sum over all bets made in a world, then it is not well-defined, for the reasons you discuss: we can’t decide how to bet without a properly-defined utility function. One approach to making it well-defined may be to use non-standard arithmetic (or surreals), but I haven’t worked that through. Another approach is to sum bets only within N Hubble volumes of the agent (assume the agent doesn’t really care about far far away bets), and then only later take the limit as N tend to infinity. This leads to SIA betting odds.
Until recently, I thought that SIA odds meant betting heavily on H3.2 in the state E0, and then reverting to an evens bet in the state E1 (so it counters the Doomsday argument). However, the more recent analysis of SIA indicates that there is still a Doomsday shift because of “great filter” arguments (a variant of Fermi’s paradox), so the betting odds in state E1 should still be weighted towards H3.1.
Basically it doesn’t look good, since every combination of utility function or SSA with or without SIA is now creating a Doomsday shift. The only remaining let out I’ve been considering is a specially-constructed reference class (as used in SSA), but it looks like that won’t work either: in Armstrong’s analysis, we don’t get to define the reference class arbitrarily, since it consists of all linked decisions. (In the UDT case, all decisions that are made by any agents anywhere applying UDT).
I don’t see why it would make more sense to speak of the probability that you are the nth observer given some theory than the probability that you are a rock or that you are a nonsentient AI. If you object that anthropic reasoning is about consciousness, you’d have to bite the bullet and say that doomsday argument-like reasoning is invalid for nonsentient computer programs.
Also, if you’re interested in anthropics, I recommend this paper by the Future of Humanity Institute’s Stuart Armstrong.
Thanks for the response.
Clearly, you can define whatever theory you like. But to be testable, the theory has to make some predictions about our observations. For this, it is not sufficient for a theory to describe in “third party” terms what events objectively happen in the world. There also has to be a “first party” (or “indexical”) component to answer the question: OK then, but what should we expect to observe?
As an analogy, you could treat the third party component as an objectively accurate map, and the first party component as a “You are here” marker placed on the map. The map is going to be pretty useless without the marker.
Similarly, if a theory of the world doesn’t have a “first party” component, then in general it is not possible to extract predictions about what we should be observing. This is especially true if the “third party” component describes a very large of infinite universe (or multiverse). Further, if the “first party” component predicts that we should be rocks or nonsentenient AIs, well then it seems that we can falsify that component straight away. This would be similar to a “you are here” marker out in the ocean somewhere, when we know we are on dry land.
Finally, just to advise, my post was not a general defence of the Doomsday Argument, but rather a discussion on whether the SIA “move” works to defuse its conclusions. It’s quite noticeable in the DA literature that lots of people are really sure that there is something wrong with the DA, and they give all sorts of different diagnoses as to where the flaw lies, but then most of these objections either turn out not to work, or to be objections against Bayesian inference full stop. There has been—as far as I can tell—a general view that the SIA does work as a way of defusing the DA, and so it is a real objection to the DA, if you accept the SIA. The discussion then diverts into whether it is legitimate to accept the SIA.
But I disagree with that view. My analysis is effectively this: “Fine, I will grant you the SIA for the sake of argument. So you now have an infinite world. But then—oh dear—on close inspection, it looks like you haven’t defused the DA after all. In fact you just made it even stronger.”
This comment made me think about anthropics, but I never got back to you about my conclusions. What I decided is that the first party component looks really important, but it is irrelevant to decision making (except insofar as our utility functions value the existence of conscious beings). For example, if one world has 100 of me and the other has only one, I might want to precommit to some strategy based on just a Solomonoff prior or similar, with no anthropic considerations. If I do that, I wouldn’t want to change my mind based on the additional knowledge that I exist. That was rather jargonny, so see some introductions to UDT or an illustrative though experiment.
I therefore disagree with your conclusion about the DA. If Omega tells me that he is going to create either world 3.1 or world 3.2, I would be surprised in some sense to find myself as one of the first beings in world 3.2, but finding out that I am one of the first 200 billion observers does not give me any reason to violate a rationally chosen precommitment. This seems odd, but the decision-theoretic considerations necessitated by the various thought experiments associated with UDT just seem to be a different sort of thing than first person experience. (I’m referring to the thought experiments specifically mentioned in the above links, in case that isn’t clear. Also, this isn’t in those links, but it might be illustrative to consider a nonsentient optimization process. Such a thing clearly does not have first-person experience and isn’t in our reference class, whatever that means, but it can still apply decision theory to achieve its goals.)
I’m not sure I get this. I think I’ve grasped the high-level point about UDT (that the epistemic probabilities strictly never update). So that if a UDT agent has a Solomonoff prior, they always use that prior to make decisions, regardless of evidence observed.
However, UDT agents have still got to bet in some cases, and they still observe evidence which can influence their bets. Suppose that each UDT agent is offered a betting slip which pays out 1 utile if the world is consistent with H3.1, and nothing otherwise. Suppose an agent has observed or remembers evidence E. How much in utiles should the agent be prepared to pay for that slip? If she pays x (< 1) utiles, then doesn’t that define a form of subjective probability P[H3.1|E] = x? And doesn’t that x vary with the evidence?
Let’s try to step it through. Suppose in the Solomonoff prior that H3.1 has a probability p31 and H3.2 has a probability p32. Suppose also that the probability of a world containing self-aware agents who have discovered UDT is pu and the probability of an infinite world with such agents is pui.
Suppose now that an agent is aware of its own existence, and has reasoned its way to UDT but doesn’t yet know anything much else about the world; it certainly doesn’t know yet how many observers there have been yet in its own civilization. Let’s call this evidence E0.
Should the agent currently pay p31 for the betting slip, or pay something different as her value of P[H3.1 | E0]? If something different, then what? (A first guess is that it is p31/pu; an alternative guess is p31/pui if the agent is effectively applying SIA). Also, at this point, how much should the agent pay for a bet that pays off if the world is infinite: would that be pui/pu or close to 1?
Now suppose the agent learns that she is the 100 billionth observer in her civilization, creating evidence E1. How much should the agent now pay for the betting slip as the value of P[H3.1| E1]? How much should the agent pay for the infinite bet?
Finally, do the answers depend at all on the form of the utility function, and on whether correct bets by other UDT agents add to that utility function? From my understanding of Stuart Armstrong’s paper, the form of the utility function does matter in general, but does UDT make any difference here? (If the utility depends on other agents’ correct bets, then we need to be especially careful in the case of worlds with infinitely many agents, since we are summing over infinitely many wins or losses).
That analysis uses standard probability theory and decision theory, but that doesn’t work in this sort of situation.
Compare this to Psy-Kosh’s non-anthropic problem. Before you are told whether you are a decider, you can see, in the normal way, that it is better to follow the strategy of choosing “nay” rather than “yea” no matter what. If you condition on finding out that you are a decider the same way that you would condition on any piece of evidence, it appears that it would be better to choose “yea”, but we can see that someone following the strategy of updating like that will get less utility, in expectation, than someone who follows the strategy of choosing “nay” in response to any evidence. Therefore, decision theory as usual leads to suboptimal outcomes in situations like this.
In your situation, we can follow the same principles to see that standard decision can fail there too, depending on your utility function. From a UDT perspective, you are choosing between the strategies “If you have a low birth rank, be willing to pay almost a full util, if necessary, for the betting slip.” and “Only buy the slip if it is reasonably priced, ie. costs < p31 utils, no matter what you observe.” and you should weigh the resulting utility differences using the prior (p31, p32) for hypotheses H3.1 and H3.2 that you assign before you observe your birth rank. A utilitarian would know, before observing their actual birth rank, that they would obtain the highest utility by following the second strategy, so updating as you suggest would give a different answer than the one that looks best in advance, in the same way that updating in the usual way in Psy-Kosh’s problem produces a suboptimal outcome. Stuart goes into a bit more detail on this, discussing more types of utility functions and doing more of the math explicitly.
You seem to be arguing here that a UDT agent should NEVER update their betting probabilities (i.e. never change the amount paid for a betting slip) regardless of evidence. This seems plain wrong to me in general e.g. imagine I’m offered a lottery ticket for $1 a few minutes after the draw and I already saw the numbers drawn; if they match the numbers on the ticket, this is a great deal and I should take it. If UDT really says that I shouldn’t, then I’m sticking with CDT!
So I don’t think that is what you are arguing (i.e. don’t update betting probabilities in general); but you are arguing not to update the betting probability in this case. Correct?
Here is another example to consider, based on other types of evidence. Hypothesis B1 is that the universe is infinite, homogeneous, isotropic and has a background radiation of 1K. Hypothesis B3 is the same, except with background radiation of 3K. These have Solomonoff prior probabilities of pb1 and pb3 respectively. A UDT agent measures the background radiation and it comes out at 3K. Does a UDT agent still pay pb1 for the betting slip which pays out if B1 is correct? (Remember that there are infinitely many UDT agents in both B1 and B3 worlds, and infinitely many agents in each case will have just made a 3K measurement, because of measurement errors in the 1K world. The agent still doesn’t know for sure which sort of world she’s in.)
I presume you will want the agent to change betting probabilities in the background radiation case (because if she doesn’t, she’s about to lose against a CDT agent who does e.g. when betting on subsequent measurements)? And if it’s right to change in that case, why isn’t it right to change in the H31/H32 case?
Yes, I do think you should update in that case. The one-sentence version of UDT (UDT1.1 more precisely) is that you precommit to the best possible strategy and then follow that strategy. The strategy is allowed to recommend updating. In fact, the agent in my previous example explicitly considered the strategy “If you have a low birth rank, be willing to pay almost a full util, if necessary, for the betting slip.” which tells the agent to make different bets depending on what they observe. This strategy was not chosen because the agent thought that it would result in a lower expected utility, but in other situations, such as the example you just presented, the optimal strategy does entail taking different actions in response to evidence. In many thought experiments, UDT adds up to exactly what you would expect.
Actually, I’m not sure whether the strategy that I selected is optimal. It would be optimal if a single world of either the type mentioned in H3.1 or the type from H3.2 existed, but we can’t sum that over an infinite number of worlds since the sum diverges. We could switch to a bounded utiliy function, but that’s a different issue entirely.
I’m pretty sure that it isn’t optimal, and for a much simpler reason than having infinitely many worlds. The strategy of “Only buy the slip if it is reasonably priced, ie. costs < p31 utils, no matter what you observe” leads to a Dutch Book. This takes a bit of explaining, so I’ll try to simplify.
First let’s suppose that the two hypotheses are the only candidates, and that in prior probability they have equal probability 1⁄2.
H3.1. Across all of space time, there are infinitely many civilizations of observers, but the mean number of observers per civilization (taking a suitable limit construction to define the mean) is 200 billion observers.
H3.2. Across all of space time, there are infinitely many civilizations of observers, but the mean number of observers per civilization (taking the same limit construction) is 200 billion trillion observers.
We’ll also suppose that both H3.1 and H3.2 imply the existence of self-aware observers who have reasoned their way to UDT (call such a being a “UDT agent”), and slightly simplify the evidence sets E0 and E1:
E0. A UDT agent is aware of its own existence, but doesn’t yet know anything much else about the world; it certainly doesn’t know yet how many observers there have been yet in its own civilization.
(If you’re reading Stuart Armstrong’s paper, this corresponds to the “ignorant rational baby stage”).
E1. A UDT agent discovers that it is among the first quadrillion (thousand trillion) observers of its civilization.
Again, we define P[X|Ei] as the utility that the agent will pay for a betting slip which pays off 1 utile in the event that hypothesis X is true. You are proposing the following (no updating):
P[H3.1 |E0] = P[H3.2|E0] = 1⁄2, P[H3.1 |E1] = P[H3.2|E1] = 1⁄2
Now, what does the agent assign to P[E1 | E0]? Imagine that the agent is facing a bet as follows. “Omega is about to tell you how many observers there have been before you in your civilization. This betting slip will pay 1 utile if that number is less than 1 quadrillion.”
It seems clear that P[H3.1 & E1 | E0] is very close to P[H3.1 | E0]. If H3.1 is true, then essentially all observers will learn that their observer-rank is less than one quadrillion (forget about the tiny tail probability for now).
It also seems clear that P[H3.2 & E1| E0] is very close to zero, since if H3.2 is true, only a miniscule fraction of observers will learn that their observer-rank is less than one quadrillion (again forget the tiny tail probability).
So to good approximation, we have betting probabilities P[E1 & H3.1 | E0] = P[E1 | E0] = P[H3.1 | E0] = 1⁄2 and P[~E1 | E0] = 1⁄2. Thus the agent should pay 1⁄2 for a betting slip which pays out 1 utile in the event E1 & H3.1. The agent should also pay 1⁄4 for a betting slip which pays out 1⁄2 utile in the event ~E1.
Now, suppose the agent learns E1. According to your proposal, the agent still has P[H3.1 | E1] = 1⁄2, so the agent should now be prepared to sell the betting slip for E1 & H3.1 for the same price that she paid for it i.e. she sells it again for 1⁄2 a utile.
Oops: the agent is now guaranteed to lose 1⁄4 utile in all circumstances, regardless of whether or not she learns E1 or ~E1. If she learns ~E1, then she pays 3⁄4 for her two bets and wins the ~E1 bet, for a net loss of 1⁄4. If she learns E1 then she loses the bet on ~E1 and her bet on H3.1 & E1 is cancelled out since she has bought and sold the slip at the same price.
Incidentally, Stuart Armstrong discusses this issue in connection with the “Adam and Eve” problem, though he doesn’t give an explicit example of the Dutch book (I had to construct one). The resolution Stuart proposes is that an agent in the E0 (“ignorant rational baby”) stage should precommit not to sell the betting slip again if she learns E1 (or, strictly, not to sell it again unless the sale price is very close to 1 utile). Since we are discussing UDT agents, no such precommitment is needed; the agent will do whatever she should have precommited to do.
In practice this means that on learning E1, the agent follows the commitment and sets her betting probability for H3.1 very close to 1. This is, of course, a Doomsday shift.
You’re still, y’know, updating. Consider each of these bets from the updateless perspective, as strategies to be willing to accept such bets.
The first bet is to “pay 1⁄2 for a betting slip which pays out 1 utile in the event E1 & H3.1”. Adopting the strategy of accepting this kind of bet would result in 1⁄2 util for an infinite number of beings and −1/2 util for an infinite number of beings if H3.1 is true and would result in −1/2 util for an infinite number of beings if H3.2 is true.
If we could aggregate the utilities here, we could just take an expectation by weighing them according to the prior (equally in this case) and accept the bet iff the result was positive. This would give consistent, un-Dutch-bookable, results; since expectations sum, the sum of three bets with nonnegative expectations must itself have a nonnegative expectation. Unfortunately, we can’t do this since, unless you come up with some weird aggregation method other than total or average for the utility function (though my language above basically presumed totalling), the utility is a divergent series and reordering divergent series changes their sums. There is no correct ordering of the people in this scenario, so there is no correct value of the expected utility.
Moving on to the second bet, “pay 1⁄4 for a betting slip which pays out 1⁄2 utile in the event ~E1”, we see that the strategy of accepting this gives +1/2 infinitely many times and −1/2 infinitely many times if H3.1 is true and it gives −1/2 infinitely many times if H3.2 is true. Again, we can’t do the sums.
Finally the third bet, rephrased as a component of a strategy, would be to sell the betting slip from the first bet back for 1⁄2 util again if E1 is observed. Presumably, this opportunity is not offered if ¬E1, so there is no need for the agent to decide what to do in this case. This gives −1/2 infinitely many times if H3.1 and +1/2 infinitely many times if H3.2. The value of 1⁄2 −1/2 ∞ + 1⁄2 1⁄2 ∞ is, of course, indeterminate, so we can again neither recommend accepting or declining this bet without a better treatment of infinities.
I’m being careful to define the expressions P[X|Ei] as the amount paid for a betting slip on X in an evidential state Ei. This is NOT the same as the agent’s credence in hypothesis X. I agree with you that credences don’t update in UDT (that’s sort of the point). However, I’m arguing that betting payments must change (“update” if you like) between the two evidential states, or else the agent will get Dutch booked.
You describe your strategy as having an infinite gain or loss in each case, so you don’t know whether it is correct (indeed you don’t know which strategy is correct for the same reason). However, earlier up in the thread I already explained that this problem will arise if an agent’s utility depends on bets won or lost by other agents. If instead, each agent has a private utility function (and there is no addition/subtraction for other agents’ bets; only for her own) then this “adding infinities” problem doesn’t arise. Under your proposed strategy (same betting payments in E0 and E1), each individual agent gets Dutch-booked and makes a guaranteed loss of 1⁄4 utile so it can’t be the optimal strategy.
What is optimal then? In the private utility case (utility is a function only of the agent’s own bets), the optimal strategy looks to be to commit to SSA betting odds (which in the simplified example means an evens bet in the state E0, and a Doomsday betting shift in the state E1).
If the agent’s utility function is an average over all bets actually made in a world (average utilitarianism) then provided we take a sensible way of defining the average, such as take the mean (betting gain—betting loss) over N Hubble volumes, then take the limit as N goes to infinity, the optimal strategy is again SSA betting odds.
If the agent’s utility function is a sum over all bets made in a world, then it is not well-defined, for the reasons you discuss: we can’t decide how to bet without a properly-defined utility function. One approach to making it well-defined may be to use non-standard arithmetic (or surreals), but I haven’t worked that through. Another approach is to sum bets only within N Hubble volumes of the agent (assume the agent doesn’t really care about far far away bets), and then only later take the limit as N tend to infinity. This leads to SIA betting odds.
Until recently, I thought that SIA odds meant betting heavily on H3.2 in the state E0, and then reverting to an evens bet in the state E1 (so it counters the Doomsday argument). However, the more recent analysis of SIA indicates that there is still a Doomsday shift because of “great filter” arguments (a variant of Fermi’s paradox), so the betting odds in state E1 should still be weighted towards H3.1.
Basically it doesn’t look good, since every combination of utility function or SSA with or without SIA is now creating a Doomsday shift. The only remaining let out I’ve been considering is a specially-constructed reference class (as used in SSA), but it looks like that won’t work either: in Armstrong’s analysis, we don’t get to define the reference class arbitrarily, since it consists of all linked decisions. (In the UDT case, all decisions that are made by any agents anywhere applying UDT).