You then assume, without justification, that “exploring the landscape of possible worlds” will not be expressed as a part of CEV, and criticize it on these grounds.
Because EY has specifically said that that must be avoided, when he describes evolution as something dangerous. I don’t think there’s any coherent way of saying both that CEV will constrain future development (which is its purpose), and that it will not prevent us from reaching some of the best optimums.
Most likely, all the best optimums lie in places that CEV is designed to keep us away from, just as trilobite CEV would keep us away from human values. So CEV is worse than random.
Most likely, all the best optimums lie in places that CEV is designed to keep us away from, just as trilobite CEV would keep us away from human values.
That a “trilobite CEV” would never lead to human values is hardly a criticism of CEV’s effectiveness. The world we have now is not “trilobite friendly”; trilobites are extinct!
CEV, as I understand it, is very weakly specified. All it says is that a developing seed AI chooses its value system after somehow taking into account what everyone would wish for, if they had a lot more time, knowledge, and cognitive power than they do have. It doesn’t necessarily mean, for example, that every human being alive is simulated, given superintelligence, and made to debate the future of the cosmos in a virtual parliament. The combination of better knowledge of reality and better knowledge of how the human mind actually works may make it extremely clear that the essence of human values, extrapolated, is XYZ, without any need for a virtual referendum, or even a single human simulation.
It is a mistake to suppose, for example, that a human-based CEV process will necessarily give rise to a civilizational value system which attaches intrinsic value to such complexities as food, sex, or sleep, and which will therefore be prejudiced against modes of being which involve none of these things. You can have a value system which attributes positive value to human beings getting those things, not because they are regarded as intrinsically good, but because entities getting what they like is regarded as intrinsically good.
If a human being is capable of proposing a value system which makes no explicit mention of human particularities at all (e.g. Ben Goertzel’s “growth, choice, and joy”), then so is the CEV process. So if the worry is that the future will be kept unnecessarily anthropomorphic, that is not a valid critique. (It might happen if something goes wrong, but we’re talking about the basic idea here, not the ways we might screw it up.)
You could say, even a non-anthropomorphic CEV might keep us away from “the best optimums”. But let’s consider what that would mean. The proposition would be that even in a civilization making the best, wisest, most informed, most open-minded choices it could make, it still might fall short of the best possible worlds. For that to be true, must it not be the case that those best possible worlds are extremely hard to “find”? And if you propose to find them by just being random, must there not be some risk of instead ending up in very bad futures? This criticism may be comparable to the criticism that rational investment is a bad idea, because you’d make much more money if you won the lottery. If these distant optima are so hard to find, even when you’re trying to find good outcomes, I don’t see how luck can be relied upon to get you there.
This issue of randomness is not absolute. One might expect a civilization with an agreed-upon value system to nonetheless conduct fundamental experiments from time to time. But if there were experiments whose outcomes might be dangerous as well as rewarding, it would be very foolish to just go ahead and do them because if we get lucky, the consequences would be good. Therefore, I do not think that unconstrained evolution can be favored over the outcomes of non-anthropomorphic CEV.
Because EY has specifically said that that must be avoided, when he describes evolution as something dangerous.
That doesn’t mean that you can’t examine possible trajectories of evolution for good things you wouldn’t have thought of yourself, just that you shouldn’t allow evolution to determine the actual future.
I don’t think there’s any coherent way of saying both that CEV will constrain future development (which is its purpose), and that it will not prevent us from reaching some of the best optimums.
I’m not sure what you mean by “constrain” here. A process that reliably reaches an optimum (I’m not saying CEV is such a process) constrains future development to reach an optimum. Any nontrivial (and non-self-undermining, I suppose; one could value the nonexistence of optimization processes or something) value system, whether “provincially human” or not, prefers the world to be constrained into more valuable states.
Most likely, all the best optimums lie in places that CEV is designed to keep us away from
I don’t see where you’ve responded to the point that CEV would incorporate whatever reasoning leads you to be concerned about this.
Because EY has specifically said that that must be avoided, when he describes evolution as something dangerous. I don’t think there’s any coherent way of saying both that CEV will constrain future development (which is its purpose), and that it will not prevent us from reaching some of the best optimums.
Most likely, all the best optimums lie in places that CEV is designed to keep us away from, just as trilobite CEV would keep us away from human values. So CEV is worse than random.
That a “trilobite CEV” would never lead to human values is hardly a criticism of CEV’s effectiveness. The world we have now is not “trilobite friendly”; trilobites are extinct!
CEV, as I understand it, is very weakly specified. All it says is that a developing seed AI chooses its value system after somehow taking into account what everyone would wish for, if they had a lot more time, knowledge, and cognitive power than they do have. It doesn’t necessarily mean, for example, that every human being alive is simulated, given superintelligence, and made to debate the future of the cosmos in a virtual parliament. The combination of better knowledge of reality and better knowledge of how the human mind actually works may make it extremely clear that the essence of human values, extrapolated, is XYZ, without any need for a virtual referendum, or even a single human simulation.
It is a mistake to suppose, for example, that a human-based CEV process will necessarily give rise to a civilizational value system which attaches intrinsic value to such complexities as food, sex, or sleep, and which will therefore be prejudiced against modes of being which involve none of these things. You can have a value system which attributes positive value to human beings getting those things, not because they are regarded as intrinsically good, but because entities getting what they like is regarded as intrinsically good.
If a human being is capable of proposing a value system which makes no explicit mention of human particularities at all (e.g. Ben Goertzel’s “growth, choice, and joy”), then so is the CEV process. So if the worry is that the future will be kept unnecessarily anthropomorphic, that is not a valid critique. (It might happen if something goes wrong, but we’re talking about the basic idea here, not the ways we might screw it up.)
You could say, even a non-anthropomorphic CEV might keep us away from “the best optimums”. But let’s consider what that would mean. The proposition would be that even in a civilization making the best, wisest, most informed, most open-minded choices it could make, it still might fall short of the best possible worlds. For that to be true, must it not be the case that those best possible worlds are extremely hard to “find”? And if you propose to find them by just being random, must there not be some risk of instead ending up in very bad futures? This criticism may be comparable to the criticism that rational investment is a bad idea, because you’d make much more money if you won the lottery. If these distant optima are so hard to find, even when you’re trying to find good outcomes, I don’t see how luck can be relied upon to get you there.
This issue of randomness is not absolute. One might expect a civilization with an agreed-upon value system to nonetheless conduct fundamental experiments from time to time. But if there were experiments whose outcomes might be dangerous as well as rewarding, it would be very foolish to just go ahead and do them because if we get lucky, the consequences would be good. Therefore, I do not think that unconstrained evolution can be favored over the outcomes of non-anthropomorphic CEV.
That doesn’t mean that you can’t examine possible trajectories of evolution for good things you wouldn’t have thought of yourself, just that you shouldn’t allow evolution to determine the actual future.
I’m not sure what you mean by “constrain” here. A process that reliably reaches an optimum (I’m not saying CEV is such a process) constrains future development to reach an optimum. Any nontrivial (and non-self-undermining, I suppose; one could value the nonexistence of optimization processes or something) value system, whether “provincially human” or not, prefers the world to be constrained into more valuable states.
I don’t see where you’ve responded to the point that CEV would incorporate whatever reasoning leads you to be concerned about this.