For what it’s worth, the simple-minded answer given by UDT is 2. If Bob’s argumentless utility computation is U(), whose internals encode both Bob’s prior and his utility, then Bob would be willing to pay Alice a penny to maximize U(), but won’t necessarily pay to maximize a version of U() that is “remixed” using Alice’s beliefs. The distinction between 2 and 3 goes away because Alice is optimizing her input-output map instead of doing Bayesian updating.
That said, I’m not sure people’s priors actually disagree that much. The difference between religion and atheism would probably disappear by Aumann agreement.
I am not convinced that religion and theism would disappear by Aumann agreement. I know at least one religious person who is actually pretty rational, but who rejects Occam’s razor. I feel like option 3 would tell me that it would be altruistic to let this person die, so that they could go to heaven.
wait, what? how does UDT make this calculation right for Alice? Bob is wrong (from all of her knowledge and all paths for her to update). Her utility is a direct mapping of the outcome of the bet—what path of communication does Bob’s expected value take to get to her?
For a non-altruist, this is clearly a chance for Alice to money-pump Bob. But the setup of the problem is that Alice gets utility from Bob’s actual money outcome, not his beliefs. Once Alice is done updating (leaving her at 2⁄3 chance of heads), that’s her belief, and it doesn’t change after that.
Her utility is a direct mapping of the outcome of the bet—what path of communication does Bob’s expected value take to get to her?
I am not quite sure what you mean about this, but I had Alice and Bob discuss at the begining so that Alice would know Bob’s probability, and I was assuming utility of the dollar was positive. That is all she needs to know Bob’s (normalized) expected value.
The distinction between 2 and 3 is still there. 2 is still the strategy that chooses the action (not the function from inputs to actions) that Bob would like you to have chosen. 3 chooses the function from inputs to actions which 3 would like you to have chosen. Those two options are different no matter what decision theory you use.
From your claim that UDT gives 2, I think you did not understand what I meant by option 2. I think that what you mean when you say option 2 with UDT is what I mean when I say option 3. Yes, UDT doesnt update, but what option 3 is supposed to mean is that you choose the option you think Bob would want if he knew what you do. i.e. choose the function from input to output that bob would want. Option 2 was really meant to ignore your input and just care about what Bob would want you to choose not given your input. Again, I trust that this is just bad communication, and with my definitions of the 3 options you meant to say UDT says 3. Let me know if you think I am wrong about your intention.
I also disagree with the claim that UDT says 3 (or 2) is better than 1. The question is about should we take Bobs utility as a term in our own utility function (and then multiply it by our probabilities), or should we take Bob’s expected utility of our action as a term in our own utility function (which already has his probability built into it). This is a question about how we think that we should allow other peoples utility function to influence our own. UDT doesn’t tell us what kind of utility functions we should have.
Again, I trust that this is just bad communication, and with my definitions of the 3 options you meant to say UDT says 3.
Yes. Sorry for parsing your post incorrectly.
The question is about should we take Bobs utility as a term in our own utility function (and then multiply it by our probabilities), or should we take Bob’s expected utility of our action as a term in our own utility function (which already has his probability built into it).
I guess the second option sounds better to me because it generalizes more easily. What if Alice and Bob have different sets of possible worlds (or “cared-about” worlds) in the first place? What if Alice can’t disentangle the definition of Bob’s U() into “probabilities” and “utilities”, or can do it in multiple ways? My simple-minded answer still works in these cases, while the “remixing” answer seems to become more complicated.
About your other comment, it seems clear that bargaining should lead to a weighted sum like the one I described, and we get a nicer theory if altruism and bargaining are described by weighted sums of the same kind. You might disagree with arguments that rely on mathematical neatness, though...
Preference should still apply to all possible situations. If idealized Bob gains control of Alice’s decision, he has access to both the action and Alice’s factual knowledge, and so the decision specifies how the action depends on that knowledge. This looks more like option 3, even though I agree that separating prior and utility might be a wrong way of formulating this.
I think the only way to formulate option 1 is by separating prior and utility. (Not your own prior and utility, but having some model of the other person’s prior and utility separately)
I agree that option 3 is prettier because it doesn’t have to do this, but is it better?
This needs a distinction between a prior that is “prior to all your knowledge” and prior that already takes into account your current knowledge and would be updated by future observations. I guess prior in the first sense could be seen as a fixed aspect of preference, while prior in the second sense reflects a situation where an action might be performed, so that it can be different in different situations with the same preference.
Thus, perfectly altruistic Alice should have a different prior in the second sense, taking into account Alice’s knowledge rather than Bob’s, but the same prior in the first sense, reflecting the same distribution of caring over possible worlds as Bob.
Bob’s prior in the first sense is not factual knowledge, it’s a description of which worlds Bob considers how important, so Alice can’t improve on it by knowing something that Bob doesn’t. A difference in priors in the first sense reflects different distributions of moral relevance associated with possibilities. When Alice knows something that Bob doesn’t, it is a statement about her priors in the second sense, not the first sense.
Thus, to the extent that Alice doesn’t assume Bob’s priors in the first sense, Alice doesn’t follow Bob’s preference, which would be a failure of perfect altruism. Alice’s prior doesn’t reflect different (or additional) knowledge, so its use would not be an improvement in the sense of Bob’s preference.
You might disagree with arguments that rely on mathematical neatness, though...
I love mathematical neatness. The fact that the answer that felt right to me and the one that felt mathematically neat were different is what motivated me to make this post. It does not seem to me that the math of bargaining and the math of altruism and the math of bargaining should look the same though. They are not that similar, and they really feel like they are maximizing different things.
Yes, these complaints about option 1 are very real, and they bother me, which makes me unsure about my answer, and is a big part of why I created this post.
However the fact that factoring Bob’s U may not be easy or possible for Alice is not a good reason to say that Alice shouldn’t try to take that action that maximizes her expectation of Bob’s Utility. It makes her job harder, but that doesn’t mean she should try to optimize something else just because it is simpler.
I prefer 1 to 3, in spite of the fact that I think 3 actually is the more aesthetically pleasing answer.
If probability is caring, what does it mean for Alice to say that Bob’s caring is wrong? It seems to me that the intuitions in favor of option 1 are strongest in the case where some sort of “objective probability” exists and Alice has more information than Bob, not different priors. But in that case, options 1 and 3 are equivalent.
If you want to build a toy example where two agents have different but reasonable priors, maybe Robin Hanson’s pre-rationality is relevant? I’m not sure.
Note that your interpretation of altruism might make Alice go to war against Bob, even if she has no wishes of her own and cares only about being altruistic toward Bob. I guess the question is what are your desiderata for altruism?
If probability is caring, what does it mean for Alice to say that Bob’s caring is wrong?
In the exact same way that with subjective morality, relative to me, other peoples claims about morality are wrong. All I meant by that is that Alice doesn’t care in the probability sense more about the world just because Bob does because relative to Alice, Bob is simply caring about the things that are not very important.
For what it’s worth, the simple-minded answer given by UDT is 2. If Bob’s argumentless utility computation is U(), whose internals encode both Bob’s prior and his utility, then Bob would be willing to pay Alice a penny to maximize U(), but won’t necessarily pay to maximize a version of U() that is “remixed” using Alice’s beliefs. The distinction between 2 and 3 goes away because Alice is optimizing her input-output map instead of doing Bayesian updating.
That said, I’m not sure people’s priors actually disagree that much. The difference between religion and atheism would probably disappear by Aumann agreement.
I am not convinced that religion and theism would disappear by Aumann agreement. I know at least one religious person who is actually pretty rational, but who rejects Occam’s razor. I feel like option 3 would tell me that it would be altruistic to let this person die, so that they could go to heaven.
wait, what? how does UDT make this calculation right for Alice? Bob is wrong (from all of her knowledge and all paths for her to update). Her utility is a direct mapping of the outcome of the bet—what path of communication does Bob’s expected value take to get to her?
For a non-altruist, this is clearly a chance for Alice to money-pump Bob. But the setup of the problem is that Alice gets utility from Bob’s actual money outcome, not his beliefs. Once Alice is done updating (leaving her at 2⁄3 chance of heads), that’s her belief, and it doesn’t change after that.
I am not quite sure what you mean about this, but I had Alice and Bob discuss at the begining so that Alice would know Bob’s probability, and I was assuming utility of the dollar was positive. That is all she needs to know Bob’s (normalized) expected value.
Also, it is not clear to me why what Bob would be willing to pay Alice to do is necessarily what Alice should do.
I disagree. UDT does not give an answer to this.
The distinction between 2 and 3 is still there. 2 is still the strategy that chooses the action (not the function from inputs to actions) that Bob would like you to have chosen. 3 chooses the function from inputs to actions which 3 would like you to have chosen. Those two options are different no matter what decision theory you use.
From your claim that UDT gives 2, I think you did not understand what I meant by option 2. I think that what you mean when you say option 2 with UDT is what I mean when I say option 3. Yes, UDT doesnt update, but what option 3 is supposed to mean is that you choose the option you think Bob would want if he knew what you do. i.e. choose the function from input to output that bob would want. Option 2 was really meant to ignore your input and just care about what Bob would want you to choose not given your input. Again, I trust that this is just bad communication, and with my definitions of the 3 options you meant to say UDT says 3. Let me know if you think I am wrong about your intention.
I also disagree with the claim that UDT says 3 (or 2) is better than 1. The question is about should we take Bobs utility as a term in our own utility function (and then multiply it by our probabilities), or should we take Bob’s expected utility of our action as a term in our own utility function (which already has his probability built into it). This is a question about how we think that we should allow other peoples utility function to influence our own. UDT doesn’t tell us what kind of utility functions we should have.
Yes. Sorry for parsing your post incorrectly.
I guess the second option sounds better to me because it generalizes more easily. What if Alice and Bob have different sets of possible worlds (or “cared-about” worlds) in the first place? What if Alice can’t disentangle the definition of Bob’s U() into “probabilities” and “utilities”, or can do it in multiple ways? My simple-minded answer still works in these cases, while the “remixing” answer seems to become more complicated.
About your other comment, it seems clear that bargaining should lead to a weighted sum like the one I described, and we get a nicer theory if altruism and bargaining are described by weighted sums of the same kind. You might disagree with arguments that rely on mathematical neatness, though...
Preference should still apply to all possible situations. If idealized Bob gains control of Alice’s decision, he has access to both the action and Alice’s factual knowledge, and so the decision specifies how the action depends on that knowledge. This looks more like option 3, even though I agree that separating prior and utility might be a wrong way of formulating this.
I think the only way to formulate option 1 is by separating prior and utility. (Not your own prior and utility, but having some model of the other person’s prior and utility separately)
I agree that option 3 is prettier because it doesn’t have to do this, but is it better?
This needs a distinction between a prior that is “prior to all your knowledge” and prior that already takes into account your current knowledge and would be updated by future observations. I guess prior in the first sense could be seen as a fixed aspect of preference, while prior in the second sense reflects a situation where an action might be performed, so that it can be different in different situations with the same preference.
Thus, perfectly altruistic Alice should have a different prior in the second sense, taking into account Alice’s knowledge rather than Bob’s, but the same prior in the first sense, reflecting the same distribution of caring over possible worlds as Bob.
Why should Alice have the same distribution of caring as Bob?
My definition of prior was in the first sense.
Bob’s prior in the first sense is not factual knowledge, it’s a description of which worlds Bob considers how important, so Alice can’t improve on it by knowing something that Bob doesn’t. A difference in priors in the first sense reflects different distributions of moral relevance associated with possibilities. When Alice knows something that Bob doesn’t, it is a statement about her priors in the second sense, not the first sense.
Thus, to the extent that Alice doesn’t assume Bob’s priors in the first sense, Alice doesn’t follow Bob’s preference, which would be a failure of perfect altruism. Alice’s prior doesn’t reflect different (or additional) knowledge, so its use would not be an improvement in the sense of Bob’s preference.
Yes, when I said 2, I actually meant 3.
I love mathematical neatness. The fact that the answer that felt right to me and the one that felt mathematically neat were different is what motivated me to make this post. It does not seem to me that the math of bargaining and the math of altruism and the math of bargaining should look the same though. They are not that similar, and they really feel like they are maximizing different things.
Yes, these complaints about option 1 are very real, and they bother me, which makes me unsure about my answer, and is a big part of why I created this post.
However the fact that factoring Bob’s U may not be easy or possible for Alice is not a good reason to say that Alice shouldn’t try to take that action that maximizes her expectation of Bob’s Utility. It makes her job harder, but that doesn’t mean she should try to optimize something else just because it is simpler.
I prefer 1 to 3, in spite of the fact that I think 3 actually is the more aesthetically pleasing answer.
If probability is caring, what does it mean for Alice to say that Bob’s caring is wrong? It seems to me that the intuitions in favor of option 1 are strongest in the case where some sort of “objective probability” exists and Alice has more information than Bob, not different priors. But in that case, options 1 and 3 are equivalent.
If you want to build a toy example where two agents have different but reasonable priors, maybe Robin Hanson’s pre-rationality is relevant? I’m not sure.
Note that your interpretation of altruism might make Alice go to war against Bob, even if she has no wishes of her own and cares only about being altruistic toward Bob. I guess the question is what are your desiderata for altruism?
In the exact same way that with subjective morality, relative to me, other peoples claims about morality are wrong. All I meant by that is that Alice doesn’t care in the probability sense more about the world just because Bob does because relative to Alice, Bob is simply caring about the things that are not very important.