I agree completely that the extrapolation process as envisioned in CEV leads to the system doing all manner of things that the original people would reject.
It is also true that maturation often leads to adult humans doing all manner of things that their immature selves would reject. And, sure, it’s possible to adopt the “Peter Pan” stance of “I don’t wanna grow up!” in response to that, though it’s hard to maintain in the face of social expectations and biological imperatives.
It is not, however, clear to me that a seven-year-old would be wise to reject puberty, or that we would, if offered a pill that ensured that we would never come to prefer anything different from what we prefer right now, be wise to collectively take it.
That extrapolation leads to something different isn’t clearly a reason to reject it.
It is not, however, clear to me that a seven-year-old would be wise to reject puberty
The difference between a seven year old and an adult is a transition into a well understood state that many have been in. The most powerful people in society are already in that state. In contrast, the transition of the AI’s extrapolation is going into a completely new state that nobody knows anything about, except possibly the AI. The analogy isn’t valid.
That extrapolation leads to something different isn’t clearly a reason to reject it.
It’s a wildcard with unknown and unknowable consequences. That’s not a good thing to have in a Friendly AI. The burden of proof should be on the people who want to include it. As I mentioned earlier, it’s not the best solution to the Fred-wants-to-murder-Steve problem, since it’s more reliable to look at Steve’s present desire to live than to hope that extrapolated-Fred doesn’t want to murder. So it isn’t needed to solve that problem. What problem does it solve?
I agree with you that puberty is a familiar, well-understood transition, whereas extrapolation is not. It’s not clear to me that reasoning from familiar cases to novel ones by analogy is invalid, but I certainly agree that reasoning by analogy doesn’t prove much of anything, and you’re entirely justified in being unconvinced by it.
I agree with you that anyone who wants to flip the switch on what I consider a FOOMing CEV-implementing AI has an enormous burden of proof to shoulder before they get my permission to do so. (Not that I expect they will care very much.)
I agree with you that if we simply want to prevent the AI from killing people, we can cause it to implement people’s desire to live; we don’t need to extrapolate Fred’s presumed eventual lack of desire-to-kill to achieve that.
My one-sentence summary of the problem CEV is intended to solve (I do not assert that it does so) is “how do we define the target condition for a superhuman environment-optimizing system in such a way that we can be confident that it won’t do the wrong thing?”
That is expanded on at great length in the Metaethics and Fun Theory sequences, if you’re interested. Those aren’t the clearest conceivable presentation, but I doubt I will do better in a comment and am not highly motivated to try.
My one-sentence summary of the problem CEV is intended to solve (I do not assert
that it does so) is “how do we define the target condition for a superhuman
environment-optimizing system in such a way that we can be confident that it won’t do
the wrong thing?”
My question was meant to be “What problem does extrapolation solve?”, not “What problem is CEV intended to solve?” To answer the former question, you’d need some example that can be solved with extrapolation that can’t easily be solved without it. I can’t presently see a reason the example should be much more complicated than the Fred-wants-to-kill-Steve example we were talking about earlier.
That is expanded on at great length in the Metaethics and Fun Theory sequences, if
you’re interested.
I might read that eventually, but not for the purpose of getting an answer to this question. I have no reason to believe the problem solved by extrapolation is so complex that one needs to read a long exposition to understand the problem. Understanding why extrapolation solves the problem might take some work, but understanding what the problem is should not. If there’s no short description of a problem that requires extrapolation to solve it, it seems likely to me that extrapolation does not solve a problem.
For example, integral calculus is required to solve the problem “What is the area under this parabola?”, given enough parameters to uniquely determine the parabola. Are you seriously saying that extrapolation is necessary but its role is more obscure than that of integral calculus?
Are you seriously saying that extrapolation is necessary but its role is more obscure than that of integral calculus?
What I said was that the putative role of extrapolation is avoiding optimizing for the wrong thing.
That’s not noticeably more complicated a sentence than “the purpose of calculus is to calculate the area under a parabola”, so I mostly think your question is rhetorically misleading.
Anyway, as I explicitly said, I’m not asserting that extrapolation solves any problem at all. I was answering (EDIT: what I understood to be) your question about what problem it’s meant to solve, and providing some links to further reading if you’re interested, which it sounds like you aren’t, which is fine.
Ah, I see. I was hoping to find an example, about as concrete as the Fred-wants-to-kill-Steve example, that someone believes actually motivates extrapolation. A use-case, as it were.
You gave the general idea behind it. In retrospect, that was a reasonable interpretation of my question.
I’m not asserting that extrapolation solves any problem at all.
Okay, so you don’t have a use case. No problem, I don’t either. Does anybody else?
I realize you haven’t been online for a few months, but yes, I do.
Humanity’s desires are not currently consistent. An FAI couldn’t satisfy them all because some of them contradict each other, like Fred’s and Steve’s in your example. There may not even be a way of averaging them out fairly or meaningfully. Either Steve lives or dies: there’s no average or middle ground and Fred is just out of luck.
However, it might be the case that human beings are similar enough that if you extrapolate everything all humans want, you get something consistent. Extrapolation is a tool to resolve inconsistencies and please both Fred and Steve.
I agree completely that the extrapolation process as envisioned in CEV leads to the system doing all manner of things that the original people would reject.
It is also true that maturation often leads to adult humans doing all manner of things that their immature selves would reject. And, sure, it’s possible to adopt the “Peter Pan” stance of “I don’t wanna grow up!” in response to that, though it’s hard to maintain in the face of social expectations and biological imperatives.
It is not, however, clear to me that a seven-year-old would be wise to reject puberty, or that we would, if offered a pill that ensured that we would never come to prefer anything different from what we prefer right now, be wise to collectively take it.
That extrapolation leads to something different isn’t clearly a reason to reject it.
The difference between a seven year old and an adult is a transition into a well understood state that many have been in. The most powerful people in society are already in that state. In contrast, the transition of the AI’s extrapolation is going into a completely new state that nobody knows anything about, except possibly the AI. The analogy isn’t valid.
It’s a wildcard with unknown and unknowable consequences. That’s not a good thing to have in a Friendly AI. The burden of proof should be on the people who want to include it. As I mentioned earlier, it’s not the best solution to the Fred-wants-to-murder-Steve problem, since it’s more reliable to look at Steve’s present desire to live than to hope that extrapolated-Fred doesn’t want to murder. So it isn’t needed to solve that problem. What problem does it solve?
I agree with you that puberty is a familiar, well-understood transition, whereas extrapolation is not. It’s not clear to me that reasoning from familiar cases to novel ones by analogy is invalid, but I certainly agree that reasoning by analogy doesn’t prove much of anything, and you’re entirely justified in being unconvinced by it.
I agree with you that anyone who wants to flip the switch on what I consider a FOOMing CEV-implementing AI has an enormous burden of proof to shoulder before they get my permission to do so. (Not that I expect they will care very much.)
I agree with you that if we simply want to prevent the AI from killing people, we can cause it to implement people’s desire to live; we don’t need to extrapolate Fred’s presumed eventual lack of desire-to-kill to achieve that.
My one-sentence summary of the problem CEV is intended to solve (I do not assert that it does so) is “how do we define the target condition for a superhuman environment-optimizing system in such a way that we can be confident that it won’t do the wrong thing?”
That is expanded on at great length in the Metaethics and Fun Theory sequences, if you’re interested. Those aren’t the clearest conceivable presentation, but I doubt I will do better in a comment and am not highly motivated to try.
My question was meant to be “What problem does extrapolation solve?”, not “What problem is CEV intended to solve?” To answer the former question, you’d need some example that can be solved with extrapolation that can’t easily be solved without it. I can’t presently see a reason the example should be much more complicated than the Fred-wants-to-kill-Steve example we were talking about earlier.
I might read that eventually, but not for the purpose of getting an answer to this question. I have no reason to believe the problem solved by extrapolation is so complex that one needs to read a long exposition to understand the problem. Understanding why extrapolation solves the problem might take some work, but understanding what the problem is should not. If there’s no short description of a problem that requires extrapolation to solve it, it seems likely to me that extrapolation does not solve a problem.
For example, integral calculus is required to solve the problem “What is the area under this parabola?”, given enough parameters to uniquely determine the parabola. Are you seriously saying that extrapolation is necessary but its role is more obscure than that of integral calculus?
What I said was that the putative role of extrapolation is avoiding optimizing for the wrong thing.
That’s not noticeably more complicated a sentence than “the purpose of calculus is to calculate the area under a parabola”, so I mostly think your question is rhetorically misleading.
Anyway, as I explicitly said, I’m not asserting that extrapolation solves any problem at all. I was answering (EDIT: what I understood to be) your question about what problem it’s meant to solve, and providing some links to further reading if you’re interested, which it sounds like you aren’t, which is fine.
Ah, I see. I was hoping to find an example, about as concrete as the Fred-wants-to-kill-Steve example, that someone believes actually motivates extrapolation. A use-case, as it were.
You gave the general idea behind it. In retrospect, that was a reasonable interpretation of my question.
Okay, so you don’t have a use case. No problem, I don’t either. Does anybody else?
I realize you haven’t been online for a few months, but yes, I do.
Humanity’s desires are not currently consistent. An FAI couldn’t satisfy them all because some of them contradict each other, like Fred’s and Steve’s in your example. There may not even be a way of averaging them out fairly or meaningfully. Either Steve lives or dies: there’s no average or middle ground and Fred is just out of luck.
However, it might be the case that human beings are similar enough that if you extrapolate everything all humans want, you get something consistent. Extrapolation is a tool to resolve inconsistencies and please both Fred and Steve.