I admit to not being super interested in the larger geopolitical context in which this discussion is embedded… but I do want to get into this bit a little more:
think as little as possible before making a commitment in your mind, because if you think more, you might conclude (via simulation or abstract reasoning) that the other player already made their commitment so now your own decision has to condition on that commitment
It’s not obvious to me why the bolded assertion follows; isn’t the point of “updatelessness” precisely that you ignore / refrain from conditioning your decision on (negative-sum) actions taken by your opponent in a way that would, if your conditioning on those actions was known in advance, predictably incentivize your opponent to take those actions? Isn’t that the whole point of having a decision theory that doesn’t give in to blackmail?
Like, yes, one way to refuse to condition on that kind of thing is to refuse to even compute it, but it seems very odd to me to assert that this is the best way to do things. At the very least, you can compute everything first, and then decide to retroactively ignore all the stuff you “shouldn’t have” computed, right? In terms of behavior this ought not provide any additional incentives to your opponent to take stupid (read: negative-sum) actions, while still providing the rest of the advantages that come with “thinking things through”… right?
and by thinking more you also make it harder for the player to conclude this about yourself
This part is more compelling in my view, but also it kind of seems… outside of decision theory’s wheelhouse? Like, yes, once you start introducing computational constraints and other real-world weirdness, things can and do start getting messy… but also, the messiness that results isn’t a reason to abandon the underlying decision theory?
For example, I could say “Imagine a crazy person really, really wants to kill you, and the reason they want to do this is that their brain is in some sense bugged; what does your decision theory say you should do in this situation?” And the answer is that your decision theory doesn’t say anything (well, anything except “this opponent is behaviorally identical to a DefectBot, so defect against them with all you have”), but that isn’t the decision theory’s fault, it’s just that you gave it an unfair scenario to start with.
It’s not obvious to me why the bolded assertion follows; isn’t the point of “updatelessness” precisely that you ignore / refrain from conditioning your decision on (negative-sum) actions taken by your opponent in a way that would, if your conditioning on those actions was known in advance, predictably incentivize your opponent to take those actions? Isn’t that the whole point of having a decision theory that doesn’t give in to blackmail?
By “has to” I didn’t mean that’s normatively the right thing to do, but rather that’s what UDT (as currently formulated) says to do. UDT is (currently) updateless with regard to physical observations (inputs from your sensors) but not logical observations (things that you compute in your mind), and nobody seems to know how to formulate a decision theory that is logically updateless (and not broken in other ways). It seems to be a hard problem as progress has been bogged down for more than 10 years.
I admit to not being super interested in the larger geopolitical context in which this discussion is embedded… but I do want to get into this bit a little more:
It’s not obvious to me why the bolded assertion follows; isn’t the point of “updatelessness” precisely that you ignore / refrain from conditioning your decision on (negative-sum) actions taken by your opponent in a way that would, if your conditioning on those actions was known in advance, predictably incentivize your opponent to take those actions? Isn’t that the whole point of having a decision theory that doesn’t give in to blackmail?
Like, yes, one way to refuse to condition on that kind of thing is to refuse to even compute it, but it seems very odd to me to assert that this is the best way to do things. At the very least, you can compute everything first, and then decide to retroactively ignore all the stuff you “shouldn’t have” computed, right? In terms of behavior this ought not provide any additional incentives to your opponent to take stupid (read: negative-sum) actions, while still providing the rest of the advantages that come with “thinking things through”… right?
This part is more compelling in my view, but also it kind of seems… outside of decision theory’s wheelhouse? Like, yes, once you start introducing computational constraints and other real-world weirdness, things can and do start getting messy… but also, the messiness that results isn’t a reason to abandon the underlying decision theory?
For example, I could say “Imagine a crazy person really, really wants to kill you, and the reason they want to do this is that their brain is in some sense bugged; what does your decision theory say you should do in this situation?” And the answer is that your decision theory doesn’t say anything (well, anything except “this opponent is behaviorally identical to a DefectBot, so defect against them with all you have”), but that isn’t the decision theory’s fault, it’s just that you gave it an unfair scenario to start with.
What, if anything, am I missing here?
By “has to” I didn’t mean that’s normatively the right thing to do, but rather that’s what UDT (as currently formulated) says to do. UDT is (currently) updateless with regard to physical observations (inputs from your sensors) but not logical observations (things that you compute in your mind), and nobody seems to know how to formulate a decision theory that is logically updateless (and not broken in other ways). It seems to be a hard problem as progress has been bogged down for more than 10 years.
Conceptual Problems with UDT and Policy Selection is probably the best article to read to get up to date on this issue, if you want a longer answer.