So it is difficult to see on what grounds Mark can object to the FAIs wishes: if it tells him something is mortally right that is what he should do.
A distinction to be made: Mark can wish differently than the AI wishes, Mark can’t morally object to the AI’s wishes (if the AI follows morality).
Exactly because morality is not the same as preferences.
A distinction to be made: Mark can wish differently than the AI wishes, Mark can’t morally object to the AI’s wishes (if the AI follows morality).
Exactly because morality is not the same as preferences.