It’s not necessarily that we don’t want an agent that never considers something to be a moral fact, but that we are safer if we designed AGI under the assumption that moral facts don’t exist. This does not necessarily mean precluding an AGI coming to believe in the existence of moral facts and almost certainly doesn’t mean giving the AGI a preference to not believe in moral facts, but that alignment schemes should not depend on moral facts existing, and thus should not depend on the ability to make normative assumptions that will be compatible with alignment.
It’s not necessarily that we don’t want an agent that never considers something to be a moral fact, but that we are safer if we designed AGI under the assumption that moral facts don’t exist. This does not necessarily mean precluding an AGI coming to believe in the existence of moral facts and almost certainly doesn’t mean giving the AGI a preference to not believe in moral facts, but that alignment schemes should not depend on moral facts existing, and thus should not depend on the ability to make normative assumptions that will be compatible with alignment.