Nonetheless, I think after further consideration I would end up substantially increasing my expectation that if you have some moderately competent Friendly AI researchers, they would apply their skills to create an Oracle AI first; and so by Conservation of Expected Evidence I am executing that update now.
This is not relevant to FAI per se, but Michael and Susan Leigh Anderson have suggested (and begun working on) just that in the field of Machine Ethics. The main contention seems to be that creating an ethical oracle is easier than creating an embodied ethical agent because you don’t need to first figure out whether the robot is an ethical patient. Then once the bugs are out, presumably the same algorithms can be applied to embodied robots.
ETA: For reference, I think the relevant paper is “Machine Metaethics” by Susan Leigh Anderson, in the volume Machine Ethics—I’m sure lukeprog has a copy.
Why would you not need to figure out if an oracle is an ethical patient? Why is there no such possibility as a sentient oracle?
The oracle gets asked questions like “Should intervention X be used by doctor D on patient P” and can tell you the correct answer to them without considering the moral status of the oracle.
If it were a robot, it would be asking questions like “Should I run over that [violin/dog/child] to save myself?” which does require considering the status of the robot.
EDIT: To clarify, it’s not that the researcher has no reason to figure out the moral status of the oracle, it’s that the oracle does not need to know its own moral status to answer its domain-specific questions.
This is not relevant to FAI per se, but Michael and Susan Leigh Anderson have suggested (and begun working on) just that in the field of Machine Ethics. The main contention seems to be that creating an ethical oracle is easier than creating an embodied ethical agent because you don’t need to first figure out whether the robot is an ethical patient. Then once the bugs are out, presumably the same algorithms can be applied to embodied robots.
ETA: For reference, I think the relevant paper is “Machine Metaethics” by Susan Leigh Anderson, in the volume Machine Ethics—I’m sure lukeprog has a copy.
The heck? Why would you not need to figure out if an oracle is an ethical patient? Why is there no such possibility as a sentient oracle?
Is this standard religion-of-embodiment stuff?
The oracle gets asked questions like “Should intervention X be used by doctor D on patient P” and can tell you the correct answer to them without considering the moral status of the oracle.
If it were a robot, it would be asking questions like “Should I run over that [violin/dog/child] to save myself?” which does require considering the status of the robot.
EDIT: To clarify, it’s not that the researcher has no reason to figure out the moral status of the oracle, it’s that the oracle does not need to know its own moral status to answer its domain-specific questions.
What if it assigned moral status to itself and then biased its answers to make its users less likely to pull its plug one day?