Do you think that a human rule lawyer, someone built to manipulate rules and regulations, could not argue there way through this, sticking with all the technical requirements but getting completely different outcomes? I know I could.
The question isn’t whether there is one solution, but whether the space of possible solutions is encompassed by acceptable morals. I would not “expect an AI to stumble preferentially on the solution we had in mind” because I am confused and do not know what the solution is, as are you and everyone else on LessWrong. However that is a separate issue from whether we can specify what a solution would look like, such as a reflective-equilibrium solution to the coherent extrapolated volition of humankind. You can write an optimizer to search for a description of CEV without actually knowing what the result will be.
It’s like saying “I want to calculate pi to the billionth digit” and writing a program to do it, then arguing that we can’t be sure the result is correct because we don’t know ahead of time what the billionth digit of pi will be. Nonsense.
The question isn’t whether there is one solution, but whether the space of possible solutions is encompassed by acceptable morals. I would not “expect an AI to stumble preferentially on the solution we had in mind” because I am confused and do not know what the solution is, as are you and everyone else on LessWrong. However that is a separate issue from whether we can specify what a solution would look like, such as a reflective-equilibrium solution to the coherent extrapolated volition of humankind. You can write an optimizer to search for a description of CEV without actually knowing what the result will be.
It’s like saying “I want to calculate pi to the billionth digit” and writing a program to do it, then arguing that we can’t be sure the result is correct because we don’t know ahead of time what the billionth digit of pi will be. Nonsense.
Whether the space of possible solutions is contained in the space of moral outcomes.
Correct.