Great post. I think a lot of the discussion around the role of coherence arguments and what we should expect a super-intelligent agent to behave like is really sloppy and I think this distinction between “coherence theorems as a self-contained mathematical result” and “coherence arguments as a normative claim about what an agent must be like on pain of shooting themselves in the foot” is an important one
The example of how an incomplete agent avoids getting Dutch-booked also seems to look very naturally like how irl agents behave imo. One way of thinking about this is also that these lotteries are a lot more “high-dimensional” than they initially look—e.g. the decision at node 2 isn’t between “B and C” but between “B and C given I just chose B in a choice between B and A and this guy is trying to rip me off”. In general the path-dependence of our bets and our meta-preferences on how our preferences are engaged with by other agents are also legitimate reasons to expect things like Dutch-booking has less normative force for actual agents IRL. Of course in a way this is maybe just making you VNM-rational after all albeit with a super weird and garbled utility function, but that’s a whole other problem with coherence arguments
Great post. I think a lot of the discussion around the role of coherence arguments and what we should expect a super-intelligent agent to behave like is really sloppy and I think this distinction between “coherence theorems as a self-contained mathematical result” and “coherence arguments as a normative claim about what an agent must be like on pain of shooting themselves in the foot” is an important one
The example of how an incomplete agent avoids getting Dutch-booked also seems to look very naturally like how irl agents behave imo. One way of thinking about this is also that these lotteries are a lot more “high-dimensional” than they initially look—e.g. the decision at node 2 isn’t between “B and C” but between “B and C given I just chose B in a choice between B and A and this guy is trying to rip me off”. In general the path-dependence of our bets and our meta-preferences on how our preferences are engaged with by other agents are also legitimate reasons to expect things like Dutch-booking has less normative force for actual agents IRL. Of course in a way this is maybe just making you VNM-rational after all albeit with a super weird and garbled utility function, but that’s a whole other problem with coherence arguments