How do coherence theorems translate to such specific claims about the “shape of cognition”; and why is grounding these theorems in “resources” a justifiable choice in this context?
It occurs to me that one plausible answer here is that cognition requires computational resources, and therefore effective cognition will generically involve trading off these resources in a way that does not reliably lose them.
But my more relevant response is that in that section I don’t see Eliezer saying that coherence theorems are the justification for his claim about the anti-naturalness of deference.
I don’t see Eliezer saying that coherence theorems are the justification for his claim about the anti-naturalness of deference.
If coherence theorems are consistent with deference being “natural”, then I’m not sure what argument Eliezer is trying to make in this post, because then couldn’t they also be consistent with other deontological cognition being natural, and therefore likely to arise in AGIs?
effective cognition will generically involve trading off these resources in a way that does not reliably lose them
In principle, maybe. In practice, if we’d been trying to predict how monkeys will evolve, what does this claim imply about human-monkey differences?
It occurs to me that one plausible answer here is that cognition requires computational resources, and therefore effective cognition will generically involve trading off these resources in a way that does not reliably lose them.
But my more relevant response is that in that section I don’t see Eliezer saying that coherence theorems are the justification for his claim about the anti-naturalness of deference.
If coherence theorems are consistent with deference being “natural”, then I’m not sure what argument Eliezer is trying to make in this post, because then couldn’t they also be consistent with other deontological cognition being natural, and therefore likely to arise in AGIs?
In principle, maybe. In practice, if we’d been trying to predict how monkeys will evolve, what does this claim imply about human-monkey differences?