As you mentionned, no axiology can be inferred from ontology alone.
Even with meta-ethical uncertainty, if we want to build an agent that takes decisions/actions, it needs some initial axiology. If you include (P) “never consider anything as a moral fact” as part of your axology, then two things might happen:
1) This assertion (P) stays in the agent without being modified
2) The agent rewrites its own axiology and modify/delete (P)
I see a problem here. If 1) holds, then it has considered (P) has a moral fact, absurd. If 2) holds, then your agent has lost the meta-ethics principle you wanted him to keep.
So maybe you wanted to put the meta-ethics uncertainty inside the ontology ? It this is what you meant, that doesn’t seem to solve the axiology problem.
It’s not necessarily that we don’t want an agent that never considers something to be a moral fact, but that we are safer if we designed AGI under the assumption that moral facts don’t exist. This does not necessarily mean precluding an AGI coming to believe in the existence of moral facts and almost certainly doesn’t mean giving the AGI a preference to not believe in moral facts, but that alignment schemes should not depend on moral facts existing, and thus should not depend on the ability to make normative assumptions that will be compatible with alignment.
As you mentionned, no axiology can be inferred from ontology alone.
Even with meta-ethical uncertainty, if we want to build an agent that takes decisions/actions, it needs some initial axiology. If you include (P) “never consider anything as a moral fact” as part of your axology, then two things might happen:
1) This assertion (P) stays in the agent without being modified
2) The agent rewrites its own axiology and modify/delete (P)
I see a problem here. If 1) holds, then it has considered (P) has a moral fact, absurd. If 2) holds, then your agent has lost the meta-ethics principle you wanted him to keep.
So maybe you wanted to put the meta-ethics uncertainty inside the ontology ? It this is what you meant, that doesn’t seem to solve the axiology problem.
It’s not necessarily that we don’t want an agent that never considers something to be a moral fact, but that we are safer if we designed AGI under the assumption that moral facts don’t exist. This does not necessarily mean precluding an AGI coming to believe in the existence of moral facts and almost certainly doesn’t mean giving the AGI a preference to not believe in moral facts, but that alignment schemes should not depend on moral facts existing, and thus should not depend on the ability to make normative assumptions that will be compatible with alignment.