Eliezer, as far as I can tell, “reflective equilibrium” just means “the AI/simulated non-sentient being can’t think of any more changes that it wants to make” so the real question is what counts as a change that it wants to make? Your answer seems to be whatever is decided by “a human library of non-introspectively-accessible circuits”. Well the space of possible circuits is huge, and “non-introspectively-accessible” certainly doesn’t narrow it down much. And (assuming that “a human library of circuits” = “a library of human circuits”) what is a “human circuit”? A neural circuit copied from a human being? Isn’t that exactly what you argued against in “Artificial Mysterious Intelligence”?
(It occurs to me that perhaps you’re describing your understanding of how human beings do moral growth and not how you plan for an AI/simulated non-sentient being to do it. But if so, that understanding seems to be similar in usefulness to “human beings use neural networks to decide how to satisfy their desires.”)
Eliezer wrote: I don’t think I’m finished with this effort, but are you unsatisfied with any of the steps I’ve taken so far? Where?
The design space for “moral growth” is just as big as the design space for “optimization” and the size of the target you have to hit in order to have a good outcome is probably just as small. More than any dissatisfaction with the specific steps you’ve taken, I don’t understand why you don’t seem to (judging from your public writings) view the former problem to be as serious and difficult as the latter one, if not more so, because there is less previous research and existing insights that you can draw from. Where are the equivalents of Bayes, von Neumann-Morgenstern, and Pearl, for example?
Eliezer, as far as I can tell, “reflective equilibrium” just means “the AI/simulated non-sentient being can’t think of any more changes that it wants to make” so the real question is what counts as a change that it wants to make? Your answer seems to be whatever is decided by “a human library of non-introspectively-accessible circuits”. Well the space of possible circuits is huge, and “non-introspectively-accessible” certainly doesn’t narrow it down much. And (assuming that “a human library of circuits” = “a library of human circuits”) what is a “human circuit”? A neural circuit copied from a human being? Isn’t that exactly what you argued against in “Artificial Mysterious Intelligence”?
(It occurs to me that perhaps you’re describing your understanding of how human beings do moral growth and not how you plan for an AI/simulated non-sentient being to do it. But if so, that understanding seems to be similar in usefulness to “human beings use neural networks to decide how to satisfy their desires.”)
Eliezer wrote: I don’t think I’m finished with this effort, but are you unsatisfied with any of the steps I’ve taken so far? Where?
The design space for “moral growth” is just as big as the design space for “optimization” and the size of the target you have to hit in order to have a good outcome is probably just as small. More than any dissatisfaction with the specific steps you’ve taken, I don’t understand why you don’t seem to (judging from your public writings) view the former problem to be as serious and difficult as the latter one, if not more so, because there is less previous research and existing insights that you can draw from. Where are the equivalents of Bayes, von Neumann-Morgenstern, and Pearl, for example?