I don’t think that’s entirely correct; SSA, for example, is a halfer position and it does exclude worlds where you don’t exist, as do many other anthropic approaches.
Personally I’m generally skeptical of averaging over agents in any utility function.
You definitely don’t have a 50% chance of dying in the sense of “experiencing dying”. In the sense of “ceasing to exist” I guess you could argue for it, but I think that it’s much more reasonable to say that both past selves continue to exist as a single future self.
Regardless, this stuff may be confusing, but it’s entirely conceivable that with the correct theory of personal identity we would have a single correct answer to each of these questions.
OK, the “you cause 1⁄10 of the policy to happen” argument is intuitively reasonable, but under that kind of argument divided responsibility has nothing to do with how many agents are subjectively indistinguishable and instead has to do with the agents who actually participate in the linked decision.
On those grounds, “divided responsibility” would give the right answer in Psy-Kosh’s non-anthropic problem. However, this also means your argument that SIA+divided = SSA+total clearly fails, because of the example I just gave before, and because SSA+total gives the wrong answer in Psy-Kosh’s non-anthropic problem but SIA+divided does not.
Ah, subjective anticipation… That’s an interesting question. I often wonder whether it’s meaningful.
As do I. But, as Manfred has said, I don’t think that being confused about it is sufficient reason to believe it’s meaningless.
As I mentioned earlier, it’s not an argument against halfers in general; it’s against halfers with a specific kind of utility function, which sounds like this: “In any possible world I value only my own current and future subjective happiness, averaged over all of the subjectively indistinguishable people who could equally be “me” right now.”
In the above scenario, there is a 1⁄2 chance that both Jack and Roger will be created, a 1⁄4 chance of only Jack, and a 1⁄4 chance of only Roger.
Before finding out who you are, averaging would lead to a 1:1 odds ratio, and so (as you’ve agreed) this would lead to a cutoff of 1⁄2.
After finding out whether you are, in fact, Jack or Roger, you have only one possible self in the TAILS world, and one possible self in the relevant HEADS+Jack/HEADS+Roger world, which leads to a 2:1 odds ratio and a cutoff of 2⁄3.
Ultimately, I guess the essence here is that this kind of utility function is equivalent to a failure to properly conditionalise, and thus even though you’re not using probabilities you’re still “Dutch-bookable” with respect to your own utility function.
I guess it could be argued that this result is somewhat trivial, but the utility function mentioned above is at least intuitively reasonable, so I don’t think it’s meaningless to show that having that kind of utility function is going to put you in trouble.
Linked decisions is also what makes the halfer paradox go away.
I don’t think linked decisions make the halfer paradox I brought up go away. Any counterintuitive decisions you make under UDT are simply ones that lead to you making a gain in a counterfactual possible worlds at the cost of a loss in actual possible worlds. However, in the instance above you’re losing both in the real scenario in which you’re Jack, and in the counterfactual one in which you turned out to be Roger.
Granted, the “halfer” paradox I raised is an argument against having a specific kind of indexical utility function (selfish utility w/ averaging over subjectively indistinguishable agents) rather than an argument against being a halfer in general. SSA, for example, would tell you to stick to your guns because you would still assign probability 1⁄2 even after you know whether you’re “Jack” or “Roger”, and thus doesn’t suffer from the same paradox. That said, due to the reference class problem, If you are told whether you’re Jack or Roger before being told everything else SSA would give the wrong answer, so it’s not like it’s any better...
To get a paradox that hits at the “thirder” position specifically, in the same way as yours did, I think you need only replace the ticket with something mutually beneficial—like putting on an enjoyable movie that both can watch. Then the thirder would double count the benefit of this, before finding out who they were.
Are you sure? It doesn’t seem to be that this would be paradoxical; since the decisions are linked you could argue that “If I hadn’t put on an enjoyable movie for Jack/Roger, Jack/Roger wouldn’t have put on an enjoyable movie for me, and thus I would be worse off”. If, on the other hand, only one agent gets to make that decision, then the agent-parts would have ceased to be subjectively indistinguishable as soon as one of them was offered the decision.
But SIA also has some issues with order of information, though it’s connected with decisions
Can you illustrate how the order of information matters there? As far as I can tell it doesn’t, and hence it’s just an issue with failing to consider counterfactual utility, which SIA ignores by default. It’s definitely a relevant criticism of using anthropic probabilities in your decisions, because failing to consider counterfactual utility results in dynamic inconsistency, but I don’t think it’s as strong as the associated criticism of SSA.
Anyway, if your reference class consists of people who have seen “this is not room X”, then “divided responsibility” is no longer 1⁄3, and you probably have to go whole UTD.
If divided responsibility is not 1⁄3, what do those words even mean? How can you claim that only two agents are responsible for the decision when it’s quite clear that the decision is a linked decision shared by three agents?
If you’re taking “divided responsibility” to mean “divide by the number of agents used as an input to the SIA-probability of the relevant world”, then your argument that SSA+total = SIA+divided boils down to this: “If, in making decisions, you (an SIA agent) arbitrarily choose to divide your utility for a world by the number of subjectively indistinguishable agents in that world in the given state of information, then you end up with the same decisions as an SSA agent!”
That argument is, of course, trivially true because the the number of agents you’re dividing by will be the ratio between the SIA odds and the SSA odds of that world. If you allow me to choose arbitrary constants to scale the utility of each possible world, then of course your decisions will not be fully specified by the probabilities, no matter what decision theory you happen to use. Besides, you haven’t even given me any reason why it makes any sense at all to measure my decisions in terms of “responsibility” rather than simply using my utility function in the first place.
On the other hand, if, for example, you could justify why it would make sense to include a notion of “divided responsibility” in my decision theory, then that argument would tell me that SSA+total responsibility must clearly be conceptually the wrong way to do things because it uses total responsibility instead.
All in all, I do think anthropic probabilities are suspect for use in a decision theory because
They result in reflective inconsistency by failing to consider counterfactuals.
It doesn’t make sense to use them for decisions when the probabilities could depend upon the decisions (as in the Absent-Minded Driver)
That said, even if you can’t use those probabilities in your decision theory there is still a remaining question of “to what degree should I anticipate X, given my state of information”. I don’t think your argument on “divided responsibility” holds up, but even if it did the question on subjective anticipation remains unanswered.
That’s not true. The SSA agents are only told about the conditions of the experiment after they’re created and have already opened their eyes.
Consequently, isn’t it equally valid for me to begin the SSA probability calculation with those two agents already excluded from my reference class?
Doesn’t this mean that SSA probabilities are not uniquely defined given the same information, because they depend upon the order in which that information is incorporated?
I think that argument is highly suspect, primarily because I see no reason why a notion of “responsibility” should have any bearing on your decision theory. Decision theory is about achieving your goals, not avoiding blame for failing.
However, even if we assume that we do include some notion of responsibility, I think that your argument is still incorrect. Consider this version of the incubator Sleeping Beauty problem, where two coins are flipped.
HH ⇒ Sleeping Beauties created in Room 1, 2, and 3
HT ⇒ Sleeping Beauty created in Room 1
TH ⇒ Sleeping Beauty created in Room 2
TT ⇒ Sleeping Beauty created in Room 3
Moreover, in each room there is a sign. In Room 1 it is equally likely to say either “This is not Room 2” or “This is not Room 3″, and so on for each of the three rooms.
Now, each Sleeping Beauty is offered a choice between two coupons; each coupon gives the specified amount to their preferred charity (by assumption, utility is proportional to $ given to charity), but only if each of them chose the same coupon. The payoff looks like this:
A ⇒ $12 if HH, $0 otherwise.
B ⇒ $6 if HH, $2.40 otherwise.
I’m sure you see where this is going, but I’ll do the math anyway.
With SIA+divided responsibility, we have
p(HH) = p(not HH) = 1⁄2
The responsibility is divided among 3 people in HH-world, and among 1 person otherwise, therefore
EU(A) = (1/2)(1/3)$12 = $2.00
EU(B) = (1/2)(1/3)$6 + (1/2)$2.40 = $2.20
With SSA+total responsibility, we have
p(HH) = 1⁄3
p(not HH) = 2⁄3
EU(A) = (1/3)$12 = $4.00
EU(B) = (1/3)$6 + (2/3)$2.40 = $3.60
So SIA+divided responsibility suggests choosing B, but SSA+total responsibility suggests choosing A.
There’s no “should”—this is a value set.
The “should” comes in giving an argument for why a human rather than just a hypothetically constructed agent might actually reason in that way. The “closest continuer” approach makes at least some intuitive sense, though, so I guess that’s a fair justification.
The halfer is only being strange because they seem to be using naive CDT. You could construct a similar paradox for a thirder if you assume the ticket pays out only for the other copy, not themselves.
I think there’s more to it than that. Yes, UDT-like reasoning gives a general answer, but under UDT the halfer is still definitely acting strange in a way that the thirder would not be.
If the ticket pays out for the other copy, then UDT-like reasoning would lead you to buy the ticket regardless of whether you know which one you are or not, simply on the basis of having a linked decision. Here’s Jack’s reasoning:
“Now that I know I’m Jack, I’m still only going to pay at most $0.50, because that’s what I precommited to do when I didn’t know who I was. However, I can’t help but think that I was somehow stupid when I made that precommitment, because now it really seems I ought to be willing to pay 2⁄3. Under UDT sometimes this kind of thing makes sense, because sometimes I have to give up utility so that my counterfactual self can make greater gains, but it seems to me that that isn’t the case here. In a counterfactual scenario where I turned out to be Roger and not Jack, I would still desire the same linked decision (x=2/3). Why, then, am I stuck refusing tickets at 55 cents?”
It appears to me that something has clearly gone wrong with the self-averaging approach here, and I think it is indicative of a deeper problem with SSA-like reasoning. I’m not saying you can’t reasonably come to the halfer conclusion for different reasons (e.g. the “closest continuer” argument), but some or many of the possible reasons can still be wrong. That being said, I think I tend to disagree with pretty much all of the reasons one could be a halfer, including average utilitarianism, the “closest continuer”, and selfish averaging.
On 1), I agree that “pre-chewing” anthropic utility functions appears to be something of a hack. My current intuition in that regard is to reject the notion of anthropic utility (although not anthropic probability), but a solid formulation of anthropics could easily convince me otherwise.
On 2), if it’s within the zone of validity then I guess that’s sufficient to call something “a correct way” of solving the problem, but if there is an equally simple or simpler approach that has a strictly broader domain of validity I don’t think you can be justified in calling it “the right way”.
That’s a reasonable point, although I still have two major criticisms of it.
What is your resolution to the confusion about how anthropic reasoning should be applied, and to the various potential absurdities that seem to come from it? Non-anthropic probabilities do not have this problem, but anthropic probabilities definitely do.
How can anthropic probability be the “right way” to solve the Sleeping Beauty problem if it lacks the universality of methods like UDT?
The strongest argument against anthropic probabilities in decision-making comes from problems like the Absent-Minded Driver, in which the probabilities depend upon your decisions.
If anthropic probabilities don’t form part of a general-purpose decision theory, and you can get the right answers by simply taking the UDT approach and going straight to optimising outcomes given the strategies you could have, what use are the probabilities?
I won’t go so far as to say they’re meaningless, but without a general theory of when and how they should be used I definitely think the idea is suspect.
OK; I agree with you that selfishness is ill-defined, and the way to actually specify a particular kind of selfishness is to specify a utility function over all possible worlds (actual and counterfactual). Moreover, the general procedure for doing this is to assign “me” or “not me” label to various entities in the possible worlds, and derive utilities for those worlds on the basis of those labels. However, I think there are some issues that still need to be resolved here.
If I don’t exist, I value the person that most closely resembles me.
This appears suspect to me. If there is no person who closely resembles you, I guess in that case you’re indifferent, right? However, what if two people are equally close to you, how do you assign utility to them in that case? Also, why should you only value people who closely resemble you if you don’t exist? If anything, wouldn’t you care about them in worlds where you do exist?
As you’ve noted, in a simple case where you only have to worry about actual worlds and not counterfactual ones, and there is only a single “me”, assigning selfish utility is a relatively straightforward task. Being indifferent about counterfactual worlds where “you” don’t exist also makes some sense from a selfish perspective, although it brings you into potential conflict with your own past self. Additionally, the constant “C” may not be quite so arbitrary in the general case—what if your decision influences the probability of your own existence? In such a situation, the value of that constant will actually matter.
However, the bigger issue that you haven’t covered is this: if there are multiple entities in the same world to which you do (or potentially could) assign the label “me”, how do you assign utility to that world?
For example, in the scenario in your post, if I assume that the person in Room 1 in the heads world can indeed be labeled as “me”, how do I assign utilities to a tails world in which I could be either one of the two created copies? It appears to me that there are two different approaches, and I think it makes sense to apply the label “selfish” to both of them. One of them would be to add utility over selves (again a “thirder” position), and another would be to average utility over selves (which is halfer-equivalent). Nor do I think that the “adding” approach is equivalent to your notion of “copy-altruism”, because under the “adding” approach you would stop caring about your copies once you figured out which one you were, whereas under copy-altruism you would continue to care.
Under those assumptions, a “halfer” would be very strange indeed, because
1) They are only willing to pay 1⁄2 for a ticket.
2) They know that they must either be Jack or Roger.
3) They know that upon finding out which one they are, regardless of whether it’s Jack or Roger, they would be willing to pay 2⁄3.
Can a similar argument be made against a selfish thirder?
First of all, I think your argument from connection of past/future selves is just a specific case of the more general argument for reflective consistency, and thus does not imply any kind of “selfishness” in and of itself. More detail is needed to specify a notion of selfishness.
I understand your argument against identifying yourself with another person who might counterfactually have been in the same cell, but the problem here is that if you don’t know how the coin actually came up you still have to assign amounts of “care” to the possible selves that you could actually be.
Let’s say that, as in my reasoning above, there are two cells, B and C; when the coin comes up tails humans are created in both cell B and cell C, but when the coin comes up heads a human is created in either cell B or cell C, with equal probability. Thus there are 3 “possible worlds”:
1) p=1/2 human in both cells
2) p=1/4 human in cell B, cell C empty
3) p=1/4 human in cell C, cell B empty
If you’re a selfish human and you know you’re in cell B, then you don’t care about world (3) at all, because there is no “you” in it. However, you still don’t know whether you’re in world (1) or (2), so you still have to “care” about both worlds. Moreover, in either world the “you” you care about is clearly the person in cell B, and so I think the only utility function that makes sense is S = $B. If you want to think about it in terms of either SSA-like or SIA-like assumptions, you get the same answer because both in world (1) and world (2) there is only a single observer who could be identified as “you”.
Now, what if you didn’t know whether you were in cell B or cell C? That’s where things are a little different. In that case, there are two observers in world (1), either of whom could be “you”. There are basically two different ways of assigning utility over the two different “yous” in world (1)---adding them together, like a total utilitarian, and averaging them, like an average utilitarian; the resulting values are x=2/3 and x=1/2 respectively. Moreover, the first approach is equivalent to SIA, and the second is equivalent to SSA.
However, the SSA answer has a property that none of the others do. If the gnome was to tell the human “you’re in cell B”, an SSA-using human would change their cutoff point from 1⁄2 to 2⁄3. This seems to be rather strange indeed, because whether the human is in cell B or in cell C is not in any way relevant to the payoff. No human with any of the other utility functions we’ve considered would change his/her answer upon being told that they are in cell B.
That’s definitely a more elegant presentation.
I’m not too surprised to hear you had already discovered this idea, since I’m familiar with the gap between research and writing speed. As someone who is not involved with MIRI, consideration of some FAI-related problems is at least somewhat disincentivized by the likelihood that MIRI already has an answer.
As for flaws, I’ll list what I can think of. First of all, there are of course some obvious design difficulties, including the difficulty of designing US in the first place, and the difficulty of choosing the appropriate way of scaling US, but those seem to be resolvable.
One point that occurs to me under the assumptions of the toy model is that decisions involving larger differences in values of UN are at the same time more dangerous and more likely to outweigh the agent’s valuation of its future corrigibility. Moreover, simply increasing the scaling of US to compensate would cause US to significantly outweigh UN in the context of smaller decisions.
An example would be that the AI decides it’s crucial to take over the world in order to “save” it, so it starts building an army of subagents to do it, and it decides that building corrigibility into those subagents is not worth the associated risk of failure.
However, it appears that this problem can still be solved by designing US correctly in the first place; a well-designed US should clearly assign greater negative weighting to larger-scale corrigibility failures than to smaller scale ones.
There’s two other questions that I can see that relate to scaling up the toy model.
How does this model extend past the three-timestep toy scenario?
Does the model remain stable under assumptions of bounded computational power? In more complex scenarios there are obvious questions of “tiling”, but I think there is a more basic issue to answer that applies even in the three-timestep case. That is, if the agent will not be able to calculate the counterfactual utility values E[U | do(.)] exactly, can we make sure that the agent’s process of estimation will avoid making systematic errors that result in pathological behaviour?
I already have a more detailed version here; see the different calcualtions for E[T] vs E[IT]. However, I’ll give you a short version. From the gnome’s perspective, the two different types of total utilitarian utility functions are:
T = total $ over both cells
IT = total $ over both cells if there’s a human in my cell, 0 otherwise.
and the possible outcomes are
p=1/4 for heads + no human in my cell
p=1/4 for heads + human in my cell
p=1/2 for tails + human in my cell.
As you can see, these two utility functions only differ when there is no human in the gnome’s cell. Moreover, by the assumptions of the problem, the utility functions of the gnomes are symmetric, and their decisions are also. UDT proper doesn’t apply to gnomes whose utility function is IT, because the function IT is different for each of the different gnomes, but the more general principle of linked decisions still applies due to the obvious symmetry between the gnomes’ situations, despite the differences in utility functions. Thus we assume a linked decision where either gnome recommends buying a ticket for $x.
The utility calculations are therefore
E[T] = (1/4)(-x) + (1/4)(-x) + (1/2)2(1-x) = 1-(3/2)x (breakeven at 2⁄3)
E[IT] = (1/4)(0) + (1/4)(-x) + (1/2)2(1-x) = 1-(5/4)x (breakeven at 4⁄5)
Thus gnomes who are indifferent when no human is present (U = IT) should precommit to a value of x=4/5, while gnomes who still care about the total $ when no human is present (U = T) should precommit to a value of x=2/3.
Note also that this is invariant under the choice of which constant value we use to represent indifference. For some constant C, the correct calculation would actually be
E[IT | buy at $x] = (1/4)(C) + (1/4)(-x) + (1/2)2(1-x) = (1/4)C + 1-(5/4)x
E[IT | don’t buy] = (1/4)(C) + (1/4)(0) + (1/2)(0) = (1/4)C
and so the breakeven point remains at x = 4⁄5
The distinction is that a theory of “unicorns” is a theory that describes how and why other people (and probably you yourself) think about unicorns, while a theory of unicorns would explain actual unicorns. The latter would clearly fail as a theory, because you’re never going to actually see a unicorn.
The same distinction doesn’t apply to Newtonian mechanics, because Newtonian mechanics is a theory of mechanics, not a theory of how people think about mechanics.
On those grounds, I think it’s quite reasonable to say that virtual particles are real, and “unicorns” are real, but unicorns are not real.
I think I can give a concise unification of my idea with Karl’s. In short, the comment in the paper that
The concerns in Section 4.2 could potentially be addressed by some form of counterfactual (rather than conditional) reasoning.
is correct, and the fix is a pretty simple one. Basically, we want the following:
In selecting a1, the agent should act as though it was indifferent between the counterfactual possibilities of shutting down and not shutting down, conditional on the same actions and observations.
In selecting a2, the agent should desire to shut down when it is required to, and desire not to shut down when it is not required to.
Despite its indifference according to (1), the agent should care about its future corrigibility (i.e. its ability to actually shut down when the button is pressed).
One utility function that satisfies these requirements, in a general form, is
U(a1, o, a2) = {
(1-p) UN(a1, o, a2) + p max_b(US(a1, o, b)), if o is not in Press
p US(a1, o, a2) + (1-p) max_b(UN(a1, o, b)), if o is in Press
}, where p is a fixed constant.
From the agent’s point of view, this is equivalent to deciding a1 based on an a priori assumption that it will shut down (maximise US) with a fixed probability p, and not shut down (maximise UN) otherwise. However, in selecting a2 it will in fact maximise either UN or US depending upon whether the shutdown button was pressed or not. As a particular example, Karl’s proposition below is equivalent to the limit of this general case as p->0. See below for further details on how/why this utility function achieves the three criteria above.
We don’t want the agent to be indifferent to the actual event of the shutdown button being pressed. In fact, what we want is indifference as to whether or not any given event is classed as being a shutdown press. Thus the expected utility of (a1, o) for o in Press should be equivalent to the expected utility of the same (a1, o) under the counterfactual assumption that o is not in Press, and vice versa. To do this, the utilities of both branches of U (the Press branch and the NotPress branch) need to be made counterfactually equivalent, even though the agent selects a2 via UN in the NotPress branch and via US in the Press branch. This is precisely what Karl’s modification achieves.
This is pretty easy to achieve by ensuring that only the UN component of the NotPress branch depends upon a2, and only the US component of the Press branch depends upon a2.
We want the agent to be incentivised to take actions that are compatible with future shutdown, which means that the agent’s selection of a1 should be influenced by US as well as UN, but without introducing perverse incentives. Fortunately, the counterfactual indifference approach already solves the perversity issue, and so all that remains is to appropriately combine the influence of both UN and US. A reasonable approach is for the agent to act as though it will have to shut down with a fixed probability p, regardless of its observations. Because p is fixed, and because the agent is indifferent between any given observation being classed as a button press or not a button press, the agent is not in any way incentivised to influence the likelihood of the shutdown button being pressed.
Ah, but then you’re talking about a theory of “unicorns” rather than a theory of unicorns.
I think there are some rather significant assumptions underlying the idea that they are “non-relevant”. At the very least, if the agents were distinguishable, I think you should indeed be willing to pay to make n higher. On the other hand, if they’re indistinguishable then it’s a more difficult question, but the anthropic averaging I suggested in my previous comments leads to absurd results.
What’s your proposal here?