I have to admit, the proliferation of terms in this discussion is making me less and less clear that I understand what was being said when you corrected me initially, despite several attempts to clarify it. So I’m going to suggest that we roll back and try this again, keeping our working vocabulary as well-defined as we can.
As I understand EY’s account:
He endorses building an optimization process (that is, a process that acts to maximize the amount of some specified target) that uses as its target the set of human terminal values (that is, the things that we want for their own sake, rather than wanting because we believe they’ll get us something else).
He also endorses building this process in such a way that it will improve itself as required so as to be able to exert superhuman optimizing power towards its target. The term “Friendly AI” refers to processes of this sort—that is, self-improving superhuman optimization processes that use as their target the set of human terminal values.
He also endorses a particular process (building a seed AI that analyzes humans) as a way of identifying the set of human terminal values. The term “CEV” (or, sometimes, “CEV(humanity)”) refers to the output of such an analysis.
He endorses all of this not only as pragmatic for our purposes, but also as the morally right thing to do. Even if there’s an equally complex species out there whose terminal values differ from ours, on EY’s account the morally right thing to do is optimize the universe for our terminal values rather than for theirs or for some compromise between the two. Members of that species might believe that humans are wrong to do so, but if so they’ll be mistaken.
I understand that you believe I’m mistaken about some or all of the above. I’m really not clear at this point on what you think is mistaken, or what you think is true instead.
Can you edit the above to reflect where you think it’s mistaken?
The only part I disagree with strongly is the language of the last point. Referring to CEV as “THE morally right thing to do” makes it seem as if it were set in stone as the guaranteed best path to creating FAI, which it isn’t. EY argues that building Friendly AI instead of just letting the chips fall where they may is the morally right thing to do, and I’d agree with that, but not that CEV specifically is the right thing to do.
One general goal point for FAI is to target outcomes “at least as good” as those which would be caused by benevolent human mind upload(s). So, the kind of “moral development” that a community of uploads would undergo should be encapsulated within a FAI. In fact, any beneficial area of the moral state space that would be accessible starting from humans or any combination of humans and tools should be accessible by a good FAI design. CEV is one such proposal towards such a design.
As I understand it, yes, the thinking is to optimize for our terminal values instead of this hypothetical alien species or some compromise of the two. However, if values among different intelligent species converge given greater intelligence, knowledge, and self-reflection, then we would expect our FAI to have goals that converge with the alien FAI. If values do not converge, then we would suppose our FAI to have different values than alien FAIs.
A “terminal value” might include carefully thinking through philosophical questions such as this and designing the best goal content possible given these considerations. So, if there are hypothetical alien values that seem “correct” (or simply sufficiently desirable from the subjective perspective) to extrapolated humanity, these values would be integrated into the CEV-output.
I agree that EY does not assert that his proposed process for defining FAI’s optimization target (that is, seed AI calculating CEV) is necessarily the best path to FAI, nor that that proposed process is particularly right. Correction accepted.
And yes, I agree that on EY’s account, given an alien species whose values converge with ours, a system that optimizes for our terminal values also optimizes for theirs.
I have to admit, the proliferation of terms in this discussion is making me less and less clear that I understand what was being said when you corrected me initially, despite several attempts to clarify it. So I’m going to suggest that we roll back and try this again, keeping our working vocabulary as well-defined as we can.
As I understand EY’s account:
He endorses building an optimization process (that is, a process that acts to maximize the amount of some specified target) that uses as its target the set of human terminal values (that is, the things that we want for their own sake, rather than wanting because we believe they’ll get us something else).
He also endorses building this process in such a way that it will improve itself as required so as to be able to exert superhuman optimizing power towards its target. The term “Friendly AI” refers to processes of this sort—that is, self-improving superhuman optimization processes that use as their target the set of human terminal values.
He also endorses a particular process (building a seed AI that analyzes humans) as a way of identifying the set of human terminal values. The term “CEV” (or, sometimes, “CEV(humanity)”) refers to the output of such an analysis.
He endorses all of this not only as pragmatic for our purposes, but also as the morally right thing to do. Even if there’s an equally complex species out there whose terminal values differ from ours, on EY’s account the morally right thing to do is optimize the universe for our terminal values rather than for theirs or for some compromise between the two. Members of that species might believe that humans are wrong to do so, but if so they’ll be mistaken.
I understand that you believe I’m mistaken about some or all of the above.
I’m really not clear at this point on what you think is mistaken, or what you think is true instead.
Can you edit the above to reflect where you think it’s mistaken?
The only part I disagree with strongly is the language of the last point. Referring to CEV as “THE morally right thing to do” makes it seem as if it were set in stone as the guaranteed best path to creating FAI, which it isn’t. EY argues that building Friendly AI instead of just letting the chips fall where they may is the morally right thing to do, and I’d agree with that, but not that CEV specifically is the right thing to do.
One general goal point for FAI is to target outcomes “at least as good” as those which would be caused by benevolent human mind upload(s). So, the kind of “moral development” that a community of uploads would undergo should be encapsulated within a FAI. In fact, any beneficial area of the moral state space that would be accessible starting from humans or any combination of humans and tools should be accessible by a good FAI design. CEV is one such proposal towards such a design.
As I understand it, yes, the thinking is to optimize for our terminal values instead of this hypothetical alien species or some compromise of the two. However, if values among different intelligent species converge given greater intelligence, knowledge, and self-reflection, then we would expect our FAI to have goals that converge with the alien FAI. If values do not converge, then we would suppose our FAI to have different values than alien FAIs.
A “terminal value” might include carefully thinking through philosophical questions such as this and designing the best goal content possible given these considerations. So, if there are hypothetical alien values that seem “correct” (or simply sufficiently desirable from the subjective perspective) to extrapolated humanity, these values would be integrated into the CEV-output.
I agree that EY does not assert that his proposed process for defining FAI’s optimization target (that is, seed AI calculating CEV) is necessarily the best path to FAI, nor that that proposed process is particularly right. Correction accepted.
And yes, I agree that on EY’s account, given an alien species whose values converge with ours, a system that optimizes for our terminal values also optimizes for theirs.
Thanks.