I think that process is a lot more likely to go well if the AI researchers working with the superintelligence are not confused or dogmatic about ethics, and have spent some time thinking about things like utilitarianism, CEV, and how to make a rational social-engineering decision between different ethical stems in the context of a particular society. So I don’t think we need to solve the problem now, but I do think we need to educate ourselves for being part of a human+AI research effort to solve it. Especially the parts that might need to be put into a final goal of an AI helping us with that. For example, CEV is usually formulated in the context of “all humans”: what’s the actual definition of a ‘human’ there? Does an upload count? Do 109 almost identical copies of the same uploaded person get 109 votes? (See my post Uploads for why the answer should be that they get 1 vote shared between them and the original biological human.)
I think that process is a lot more likely to go well if the AI researchers working with the superintelligence are not confused or dogmatic about ethics, and have spent some time thinking about things like utilitarianism, CEV, and how to make a rational social-engineering decision between different ethical stems in the context of a particular society. So I don’t think we need to solve the problem now, but I do think we need to educate ourselves for being part of a human+AI research effort to solve it. Especially the parts that might need to be put into a final goal of an AI helping us with that. For example, CEV is usually formulated in the context of “all humans”: what’s the actual definition of a ‘human’ there? Does an upload count? Do 109 almost identical copies of the same uploaded person get 109 votes? (See my post Uploads for why the answer should be that they get 1 vote shared between them and the original biological human.)