To ask for decisions to be coherent, there need to be multiple possible situations in which decisions could be made, coherently across these situations or not. A UDT agent that picks a policy faces a single decision in a single possible situation. There is nothing else out there for the decision in this situation to be coherent with.
The options offered for the decision could be interpreted as lotteries over outcomes, but there is still only one decision to pick one lottery among them all, instead of many situations where the decision is to pick among a particular smaller selection of lotteries, different in each situation. So asking for coherence means asking what the updateless agent would do if most policies could be suddenly prohibited just before the decision (but after its preference is settled), if it were to update on the fact that only particular policies remained as options, which is not what actually happens.
I am not sure if there is any disagreement in this comment. What you say sounds right to me. I agree that UDT does not really set us up to want to talk about “coherence” in the first place, which makes it weird to have it be formalized in term of expected utility maximization.
This does not make me think intelligent/rational agents will/should converge to having utility.
I think coherence of unclear kind is an important principle that needs a place in any decision theory, and it motivates something other than pure updatelessness. I’m not sure how your argument should survive this. The perspective of expected utility and the perspective of updatelessness both have glaring flaws, respectively unwarranted updatefulness and lack of a coherence concept. They can’t argue against each other in their incomplete forms. Expected utility is no more a mistake than updatelessness.
To ask for decisions to be coherent, there need to be multiple possible situations in which decisions could be made, coherently across these situations or not. A UDT agent that picks a policy faces a single decision in a single possible situation. There is nothing else out there for the decision in this situation to be coherent with.
The options offered for the decision could be interpreted as lotteries over outcomes, but there is still only one decision to pick one lottery among them all, instead of many situations where the decision is to pick among a particular smaller selection of lotteries, different in each situation. So asking for coherence means asking what the updateless agent would do if most policies could be suddenly prohibited just before the decision (but after its preference is settled), if it were to update on the fact that only particular policies remained as options, which is not what actually happens.
I am not sure if there is any disagreement in this comment. What you say sounds right to me. I agree that UDT does not really set us up to want to talk about “coherence” in the first place, which makes it weird to have it be formalized in term of expected utility maximization.
This does not make me think intelligent/rational agents will/should converge to having utility.
I think coherence of unclear kind is an important principle that needs a place in any decision theory, and it motivates something other than pure updatelessness. I’m not sure how your argument should survive this. The perspective of expected utility and the perspective of updatelessness both have glaring flaws, respectively unwarranted updatefulness and lack of a coherence concept. They can’t argue against each other in their incomplete forms. Expected utility is no more a mistake than updatelessness.