The way I see it, the only correct value to align any AI to is not the arbitrary values of humans-in-general, assuming such a thing even exists, but rather the libertarian principle of self-ownership / non-aggression. The perfect super-AI would have no desire or purpose other than to be the “king” of an anarcho-monarchist world state and rigorously enforce contracts (probably with the aid of human, uplift, etc interpreters, judges, and juries stipulated in the contracts themselves, so that the AI does not have to make decisions about what is reasonable), including a basic social contract, binding on all sentient beings, that if they possess the capacity for moral reasoning, they are required to respect certain fundamental rights of all other sentient beings. (This would include obvious things like not eating them.) It would, essentially, be a sentient law court (and police force, so that it can recognize violations and take appropriate action), in which anything that has consciousness has its rights protected. For a super-AI to be anything other than that is asking for trouble.
The way I see it, the only correct value to align any AI to is not the arbitrary values of humans-in-general, assuming such a thing even exists, but rather the libertarian principle of self-ownership / non-aggression. The perfect super-AI would have no desire or purpose other than to be the “king” of an anarcho-monarchist world state and rigorously enforce contracts (probably with the aid of human, uplift, etc interpreters, judges, and juries stipulated in the contracts themselves, so that the AI does not have to make decisions about what is reasonable), including a basic social contract, binding on all sentient beings, that if they possess the capacity for moral reasoning, they are required to respect certain fundamental rights of all other sentient beings. (This would include obvious things like not eating them.) It would, essentially, be a sentient law court (and police force, so that it can recognize violations and take appropriate action), in which anything that has consciousness has its rights protected. For a super-AI to be anything other than that is asking for trouble.