Presumably, values will evolve differently depending on future contingencies. For example, a future with a world government that imposes universal birth control to limit population growth would probably evolve different values compared to a future that has no such global Singleton. Do you agree, and if so do you think the values evolved in different possible futures are all equivalent as far as you are concerned? If not, what criteria are you using to judge between them?
ETA: Can you explain John Holland’s theorems, or at least link to the book you’re talking about (Wikipedia says he wrote three). If you think allowing values to evolve is the right thing to do, I’m surprised you haven’t put more effort into making a case for it, as opposed to just criticizing SI’s plan.
Probably Adaptation in Natural and Artificial Systems. Here’s Holland’s most famous theorem in the area. It doesn’t suggest genetic algorithms make for some kind of optimal search—indeed, classical genetic algorithms are a pretty stupid sort of search.
That is the book. I”m referring to the entire contents of chapters 5-7. The schema theorem is used in chapter 7, but it’s only part of the entire argument, which does show that genetic algorithms approach optimal distribution of trials among the different possibilities, for a specific definition of optimal, which is not easy to parse out of Holland’s book, due to his failure to give an overview or decent summary of what he is doing. It doesn’t say anything about other forms of search that proceed other than by taking a big set of possible answers, which give stochastic results when tested, and allocating trials among them.
CEV is not any old set of evolved values. It is the optimal set of evolved values; the set you get when everything goes exactly right. Of your two proposed futures, one of them is a better approximation to this than the other; I just can’t say which one, at this time, because of lack of computational power. That’s what we want a FAI for. :)
Instead of pushing Phil to accept the entirety of your position at once, it seems better to introduce some doubt first: Is it really very hard to do better than to just not interfere? If I have other values besides evolution, should I give them up so quickly?
Also, if Phil has already thought a lot about these questions and thinks he is justified in being pretty certain about his answers, then I’d be genuinely curious what his reasons are.
What criteria does the CEV-calculator use to choose among those options? I agree that significant computational power is also required, but it’s not sufficient.
If we were able to formally specify the algorithm by which a CEV calculator should extrapolate our values, we would already have solved the Friendliness problem; your query is FAI-complete. But informally, we can say that the CEV evaluates by whatever values it has at a given step in its algorithm, and that the initial values are the ones held by the programmers.
The problem with this kind of reasoning (as the OP makes plain) is that there’s no good reason to think such CEV maximization is even logically possible. Not only do we not have a solution, we don’t have a well-defined problem.
Presumably, values will evolve differently depending on future contingencies. For example, a future with a world government that imposes universal birth control to limit population growth would probably evolve different values compared to a future that has no such global Singleton. Do you agree, and if so do you think the values evolved in different possible futures are all equivalent as far as you are concerned? If not, what criteria are you using to judge between them?
ETA: Can you explain John Holland’s theorems, or at least link to the book you’re talking about (Wikipedia says he wrote three). If you think allowing values to evolve is the right thing to do, I’m surprised you haven’t put more effort into making a case for it, as opposed to just criticizing SI’s plan.
Probably
Adaptation in Natural and Artificial Systems
. Here’s Holland’s most famous theorem in the area. It doesn’t suggest genetic algorithms make for some kind of optimal search—indeed, classical genetic algorithms are a pretty stupid sort of search.That is the book. I”m referring to the entire contents of chapters 5-7. The schema theorem is used in chapter 7, but it’s only part of the entire argument, which does show that genetic algorithms approach optimal distribution of trials among the different possibilities, for a specific definition of optimal, which is not easy to parse out of Holland’s book, due to his failure to give an overview or decent summary of what he is doing. It doesn’t say anything about other forms of search that proceed other than by taking a big set of possible answers, which give stochastic results when tested, and allocating trials among them.
CEV is not any old set of evolved values. It is the optimal set of evolved values; the set you get when everything goes exactly right. Of your two proposed futures, one of them is a better approximation to this than the other; I just can’t say which one, at this time, because of lack of computational power. That’s what we want a FAI for. :)
Instead of pushing Phil to accept the entirety of your position at once, it seems better to introduce some doubt first: Is it really very hard to do better than to just not interfere? If I have other values besides evolution, should I give them up so quickly?
Also, if Phil has already thought a lot about these questions and thinks he is justified in being pretty certain about his answers, then I’d be genuinely curious what his reasons are.
I misread the nesting, and responded as though your comment were a critique of CEV, rather than Phil’s objection to CEV. So I talked a bit past you.
But you’re evading Wei_Dai’s question here.
What criteria does the CEV-calculator use to choose among those options? I agree that significant computational power is also required, but it’s not sufficient.
If we were able to formally specify the algorithm by which a CEV calculator should extrapolate our values, we would already have solved the Friendliness problem; your query is FAI-complete. But informally, we can say that the CEV evaluates by whatever values it has at a given step in its algorithm, and that the initial values are the ones held by the programmers.
The problem with this kind of reasoning (as the OP makes plain) is that there’s no good reason to think such CEV maximization is even logically possible. Not only do we not have a solution, we don’t have a well-defined problem.
(nods) Fair enough. I don’t especially endorse that, but at least it’s cogent.