I think that adding new types of systems and agents to the universe changes the optimal “applied ethics” in the situation (I wrote about this here, in the “PS.”, the last paragraph), so they only hope for the discriminator to be 1) a general intelligence; 2) using a scale-free, naturalistic theory of ethics as a theoretical discipline for evaluating any applied ethics theories in any situations and contexts.
Also, hopefully, the “least wrong” scale-free ethics is “aligned” with humans, in the sense that it “saves” us from oblivion. For example, a version of theoretical scale-free ethics could just favour increasing the amount of consciousness in the universe while destroying as little existing consciousness as possible. Let’s say something like IIT is right. So the best plan that AGI should come up with is to engineer some mind upload scheme for humans and integrate our consciousnesses together, to form a single planetary-scale mega-consciousness (which must be more ethically valuable precisely because of the integration; by the same token, a brain is more ethically valuable than all neurons when they are isolated).
I think that adding new types of systems and agents to the universe changes the optimal “applied ethics” in the situation (I wrote about this here, in the “PS.”, the last paragraph), so they only hope for the discriminator to be 1) a general intelligence; 2) using a scale-free, naturalistic theory of ethics as a theoretical discipline for evaluating any applied ethics theories in any situations and contexts.
Also, hopefully, the “least wrong” scale-free ethics is “aligned” with humans, in the sense that it “saves” us from oblivion. For example, a version of theoretical scale-free ethics could just favour increasing the amount of consciousness in the universe while destroying as little existing consciousness as possible. Let’s say something like IIT is right. So the best plan that AGI should come up with is to engineer some mind upload scheme for humans and integrate our consciousnesses together, to form a single planetary-scale mega-consciousness (which must be more ethically valuable precisely because of the integration; by the same token, a brain is more ethically valuable than all neurons when they are isolated).