I don’t see Eliezer saying that coherence theorems are the justification for his claim about the anti-naturalness of deference.
If coherence theorems are consistent with deference being “natural”, then I’m not sure what argument Eliezer is trying to make in this post, because then couldn’t they also be consistent with other deontological cognition being natural, and therefore likely to arise in AGIs?
effective cognition will generically involve trading off these resources in a way that does not reliably lose them
In principle, maybe. In practice, if we’d been trying to predict how monkeys will evolve, what does this claim imply about human-monkey differences?
If coherence theorems are consistent with deference being “natural”, then I’m not sure what argument Eliezer is trying to make in this post, because then couldn’t they also be consistent with other deontological cognition being natural, and therefore likely to arise in AGIs?
In principle, maybe. In practice, if we’d been trying to predict how monkeys will evolve, what does this claim imply about human-monkey differences?