The main opposition appears to be between updatelessness and expected utility. Actions that make up updateless behavior result from a single decision, and can lack coherence. This is basically what the thought experiments about counterfactuals are demonstrating, actions that are individually clearly at odds with agent’s purported utility function, and only become correct as elements of a larger policy. But it gets worse, for expected utility is itself a product of the assumption of coherence between decisions! When an agent is acting updatelessly, it only makes a single decision to choose that single updateless policy, and there is no other decision for this one to be coherent with, so no reason at all for preference to have the form of expected utility. When the updateless point of view is internalized deeply enough, it starts to seem like the whole idea of expected utility has no foundation behind it. And that’s fair, for updateless decisions it doesn’t.
But some kind of coherence between actions and resulting preference that is consistent in some sense (even if it’s not necessarily expected utility specifically) intuitively seems like a natural principle in its own right, putting the idea of updatelessness into question. The incoherent actions of an updateless agent are bound by a single updateless decision jointly directing all of them, but coherent actions of an updateful agent are bound by some different kind of shared identity. If updateful actions shouldn’t be attributed to a single agent, there is still motivation to introduce an entity that encompasses them. This entity is then an origin of consistent preference, but this entity is not the scope of optimization by a single updateless decision determining all actions.
The main opposition appears to be between updatelessness and expected utility. Actions that make up updateless behavior result from a single decision, and can lack coherence. This is basically what the thought experiments about counterfactuals are demonstrating, actions that are individually clearly at odds with agent’s purported utility function, and only become correct as elements of a larger policy. But it gets worse, for expected utility is itself a product of the assumption of coherence between decisions! When an agent is acting updatelessly, it only makes a single decision to choose that single updateless policy, and there is no other decision for this one to be coherent with, so no reason at all for preference to have the form of expected utility. When the updateless point of view is internalized deeply enough, it starts to seem like the whole idea of expected utility has no foundation behind it. And that’s fair, for updateless decisions it doesn’t.
But some kind of coherence between actions and resulting preference that is consistent in some sense (even if it’s not necessarily expected utility specifically) intuitively seems like a natural principle in its own right, putting the idea of updatelessness into question. The incoherent actions of an updateless agent are bound by a single updateless decision jointly directing all of them, but coherent actions of an updateful agent are bound by some different kind of shared identity. If updateful actions shouldn’t be attributed to a single agent, there is still motivation to introduce an entity that encompasses them. This entity is then an origin of consistent preference, but this entity is not the scope of optimization by a single updateless decision determining all actions.