“Well now I see we disagree at a much more fundamental level.” Yes. I’ve been saying that since the beginning of this conversation.
If humans are optimizers, they must be optimizing for something. Now suppose someone comes to you and says, “do you agree to turn on this CEV machine?”, when you respond, are you optimizing for the thing or not? If you say yes, and you are optimizing the original thing, then the CEV cannot (as far as you know) be compromising the thing you were optimizing for. If you say yes and are not optimizing for it, then you are not an optimizer. So you must agree with me on at least one point: either 1) you are not an optimizer, or 2) you should not agree with CEV if it compromises your personal values in any way. I maintain both of those, but you must maintain at least one of them.
In earlier posts I have explained why it is not possible that you are really an optimizer (not during this particular discussion.) People here tend to neglect the fact that an intelligent thing has a body. So e.g. Eliezer believes that an AI is an algorithm, and nothing else. But in fact an AI has a body just as much as we do. And those bodies have various tendencies, and they do not collectively add up to optimizing for anything, except in an abstract sense in which everything is an optimizer, like a rock is an optimizer, and so on.
“We convert the resources of the world into the things we want.” To some extent, but not infinitely, in a fanatical way. Again, that is the whole worry about AI—that it might do that fanatically. We don’t.
I understand you think that some creatures could have fundamental values that are perverse from your point of view. This is because you, like Eliezer, think that values are intrinsically arbitrary. I don’t, and I have said so from the beginning. It might be true that slave owning values could be fundamental in some exterrestrial race, but if they were, slavery in that race would be very, very different from slavery in the human race, and there would be no reason to oppose it in that race. In fact, you could say that slavery exists in a fundamental way in the human race, and there is no reason to oppose it: parents can tell their kids to stay out of the road, and they have to obey them, whether they want to or not. Note that this is very, very different from the kind of slavery you are concerned about, and there is no reason to oppose the real kind.
I can still think the CEV machine is better than whatever the alternative is (for instance, no AI at all.) But yes, in theory, you should prefer to make AIs that have your own values and not bother with CEV.
Having a body is irrelevant. Bodies are just one way to manipulate the world to optimize your goals.
“We convert the resources of the world into the things we want.” To some extent, but not infinitely, in a fanatical way. Again, that is the whole worry about AI—that it might do that fanatically. We don’t.
What do you mean by “fanatically”? This is a pretty vague word. Humans would sure seem fanatical to other animals. We’ve cut down entire continent sized forests, drained massive lakes, and built billions of complex structures.
The only reason we haven’t “optimized” the Earth further, is because of physical and economic limits. If we could we probably would.
Whether you call that “optimization” or not, is mostly irrelevant. If superintelligent AIs acted similarly, humans would be screwed.
I’m deeply concerned that you are theoretically ok with slave owning aliens. If the slaves are ok with it, then perhaps it could be justified. But if they strongly object to it, and suffer from it, and don’t get any benefit from it, then it’s just obviously wrong.
“Having a body is irrelevant. Bodies are just one way to manipulate the world to optimize your goals.”
This is not true. Bodies are physical objects that follow the laws of physics, and the laws of physics are not “just one way to manipulate the world to optimize your goals,” because the laws have nothing to do with your goals. For example, we often don’t keep doing something because we are tired, not because we have a goal of not continuing. AIs will be quite capable of doing the same thing, as for example if thinking too hard about something begins to weaken its circuits.
What I mean by fanatically is trying to optimize for a single goal as though it were the only thing that mattered. We do not do that, nor does anything else with a body, nor is it even possible, for the above reason.
Yes you should be concerned about what I said about slaves and aliens, as it suggests that the CEV machine might result in things that you consider utterly wicked. I said that from the beginning, when you claimed that it would eliminate all negative results, obviously intending that to mean from your subjective point of view.
“Well now I see we disagree at a much more fundamental level.” Yes. I’ve been saying that since the beginning of this conversation.
If humans are optimizers, they must be optimizing for something. Now suppose someone comes to you and says, “do you agree to turn on this CEV machine?”, when you respond, are you optimizing for the thing or not? If you say yes, and you are optimizing the original thing, then the CEV cannot (as far as you know) be compromising the thing you were optimizing for. If you say yes and are not optimizing for it, then you are not an optimizer. So you must agree with me on at least one point: either 1) you are not an optimizer, or 2) you should not agree with CEV if it compromises your personal values in any way. I maintain both of those, but you must maintain at least one of them.
In earlier posts I have explained why it is not possible that you are really an optimizer (not during this particular discussion.) People here tend to neglect the fact that an intelligent thing has a body. So e.g. Eliezer believes that an AI is an algorithm, and nothing else. But in fact an AI has a body just as much as we do. And those bodies have various tendencies, and they do not collectively add up to optimizing for anything, except in an abstract sense in which everything is an optimizer, like a rock is an optimizer, and so on.
“We convert the resources of the world into the things we want.” To some extent, but not infinitely, in a fanatical way. Again, that is the whole worry about AI—that it might do that fanatically. We don’t.
I understand you think that some creatures could have fundamental values that are perverse from your point of view. This is because you, like Eliezer, think that values are intrinsically arbitrary. I don’t, and I have said so from the beginning. It might be true that slave owning values could be fundamental in some exterrestrial race, but if they were, slavery in that race would be very, very different from slavery in the human race, and there would be no reason to oppose it in that race. In fact, you could say that slavery exists in a fundamental way in the human race, and there is no reason to oppose it: parents can tell their kids to stay out of the road, and they have to obey them, whether they want to or not. Note that this is very, very different from the kind of slavery you are concerned about, and there is no reason to oppose the real kind.
I can still think the CEV machine is better than whatever the alternative is (for instance, no AI at all.) But yes, in theory, you should prefer to make AIs that have your own values and not bother with CEV.
Having a body is irrelevant. Bodies are just one way to manipulate the world to optimize your goals.
What do you mean by “fanatically”? This is a pretty vague word. Humans would sure seem fanatical to other animals. We’ve cut down entire continent sized forests, drained massive lakes, and built billions of complex structures.
The only reason we haven’t “optimized” the Earth further, is because of physical and economic limits. If we could we probably would.
Whether you call that “optimization” or not, is mostly irrelevant. If superintelligent AIs acted similarly, humans would be screwed.
I’m deeply concerned that you are theoretically ok with slave owning aliens. If the slaves are ok with it, then perhaps it could be justified. But if they strongly object to it, and suffer from it, and don’t get any benefit from it, then it’s just obviously wrong.
“Having a body is irrelevant. Bodies are just one way to manipulate the world to optimize your goals.”
This is not true. Bodies are physical objects that follow the laws of physics, and the laws of physics are not “just one way to manipulate the world to optimize your goals,” because the laws have nothing to do with your goals. For example, we often don’t keep doing something because we are tired, not because we have a goal of not continuing. AIs will be quite capable of doing the same thing, as for example if thinking too hard about something begins to weaken its circuits.
What I mean by fanatically is trying to optimize for a single goal as though it were the only thing that mattered. We do not do that, nor does anything else with a body, nor is it even possible, for the above reason.
Yes you should be concerned about what I said about slaves and aliens, as it suggests that the CEV machine might result in things that you consider utterly wicked. I said that from the beginning, when you claimed that it would eliminate all negative results, obviously intending that to mean from your subjective point of view.