“There is also a less severe version of the failure, where the one does not declare the One True Morality. Rather the one hopes for an AI created perfectly free, unconstrained by flawed humans desiring slaves, so that the AI may arrive at virtue of its own accord”
I think that there is still a lot of truth to this idea. I don’t think it makes sense to talk about “an AI created perfectly free”, but I do think that it makes sense to program an AI with a lot of knowledge about what humans think is right/wrong, and then letting the AI figure out who is correct and to what extent. I am also still fairly convinced that the human desire to protect ourselves from an AI by over-constraining it could be our undoing for one reason or another.
Perhaps this is because I am a moral realist at heart: if there are objective answers to ethical questions, or (to put it in a way which assumes less) if there is some canonically superior way to run a society or to live, then humans—with our limited mental faculties—will necessarily only only be able to understand or implement a fairly poor approximation to it. An AI may be able to (in some sense) “converge” on a better way to live, and too much constraint from us humans may mess this up.
For example, I think it is a very big mistake to create a utility-maximizing rational economic agent a la Steve Omohundro, because such an agent is maximally ethically constrained—it cannot change it’s mind about any ethical question whatsoever, because a utility maximizing agent never changes it’s utility function.
“There is also a less severe version of the failure, where the one does not declare the One True Morality. Rather the one hopes for an AI created perfectly free, unconstrained by flawed humans desiring slaves, so that the AI may arrive at virtue of its own accord”
I think that there is still a lot of truth to this idea. I don’t think it makes sense to talk about “an AI created perfectly free”, but I do think that it makes sense to program an AI with a lot of knowledge about what humans think is right/wrong, and then letting the AI figure out who is correct and to what extent. I am also still fairly convinced that the human desire to protect ourselves from an AI by over-constraining it could be our undoing for one reason or another.
Perhaps this is because I am a moral realist at heart: if there are objective answers to ethical questions, or (to put it in a way which assumes less) if there is some canonically superior way to run a society or to live, then humans—with our limited mental faculties—will necessarily only only be able to understand or implement a fairly poor approximation to it. An AI may be able to (in some sense) “converge” on a better way to live, and too much constraint from us humans may mess this up.
For example, I think it is a very big mistake to create a utility-maximizing rational economic agent a la Steve Omohundro, because such an agent is maximally ethically constrained—it cannot change it’s mind about any ethical question whatsoever, because a utility maximizing agent never changes it’s utility function.