The key is recognizing that the preference itself is completely independent from rationality or intelligence.
The orthogonality thesis is also for human beings.
I don’t think so; even if it applies to the subset of hypothetical superintelligences that factor neatly into beliefs and values, humans don’t seem to factorize this way (see Obliqueness Thesis, esp. argument from brain messiness).
The orthogonality thesis is also for human beings.
I don’t think so; even if it applies to the subset of hypothetical superintelligences that factor neatly into beliefs and values, humans don’t seem to factorize this way (see Obliqueness Thesis, esp. argument from brain messiness).