No, I would not be okay with it.
I don’t terminally value CEV. I think it would be instrumentally valuable, because scenarios where everyone wants to torture a few people are not that likely. I would prefer that only my own extrapolated utility function controlled the universe. Unlike Eliezer Yudkowsky, I don’t care that much about not being a jerk. But that is not going to happen.
If this detail from the original paper still stands, the CEV is allowed to modify the extrapolating process. So if there was the threat of everyone having to race to clone themselves as much as possible for more influence, it might modify itself to give clones less weight, or prohibit cloning.
So if there was the threat of everyone having to race to clone themselves as much as possible for more influence, it might modify itself to give clones less weight, or prohibit cloning
Prohibiting these things, and CEV self-modifying in general, means optimizing for certain values or a certain outcome. Where do these values come from? From the CEV’s programmers. But if you let certain predetermined values override the (unknown) CEV-extrapolated values, how do you make these choices, and where do you draw the line?
I mean that the CEV extrapolated from the entire population before they start a clone race could cause that self-modification or prohibition, not something explicitly put in by the programmers.
No, I would not be okay with it. I don’t terminally value CEV. I think it would be instrumentally valuable, because scenarios where everyone wants to torture a few people are not that likely. I would prefer that only my own extrapolated utility function controlled the universe. Unlike Eliezer Yudkowsky, I don’t care that much about not being a jerk. But that is not going to happen. If this detail from the original paper still stands, the CEV is allowed to modify the extrapolating process. So if there was the threat of everyone having to race to clone themselves as much as possible for more influence, it might modify itself to give clones less weight, or prohibit cloning.
Prohibiting these things, and CEV self-modifying in general, means optimizing for certain values or a certain outcome. Where do these values come from? From the CEV’s programmers. But if you let certain predetermined values override the (unknown) CEV-extrapolated values, how do you make these choices, and where do you draw the line?
I mean that the CEV extrapolated from the entire population before they start a clone race could cause that self-modification or prohibition, not something explicitly put in by the programmers.