What Steven said. If course, preference is not about preserving something we don’t want preserved, such as satisfaction of human drives as they currently are. Specifying the ways in which human values could grow is not vacuous, as some ways in which values could develop are better than others.
Back to definitions, human preference is whatever you (being a human) happen to prefer, on reflection. If you are right and stopping moral growth is undesirable (I agree), then by definition stopping moral growth is not part of human preference. And so on. Human preference is the specification at the top of meta, that describes all possible considerations about the ways in which all other relevant developments should happen.
I don’t think there is a top to meta; and if there is, there’s nothing human about it.
You are still speaking as if there were one privileged, appropriate level of analysis for values. In fact, as with everything expressed by human language, there are different levels of abstraction, that are appropriate in different circumstances.
The question of how meta to go, depends on the costs, the benefits, the certainty of the analysis, and other factors.
The question of how to go meta cannot be made independently of the very sorts of values that the friendly AI and CEV are themselves supposed to arbitrate between. There is no way to get outside the system and do it objectively.
human preference is whatever you (being a human) happen to prefer, on reflection.
That is not a definition, no matter how many times it’s been repeated. It’s a tautology. That is side-stepping the issue. You need to either start being specific about values, or stop asking people to respect Friendly AI and coherent extrapolated volition as if they were coherent ideas. I’ve been waiting years for an explanation, and yet these things are still developed only to the level of precision of a dope-fueled dormitory rap session. Yet, somehow, instead of being dismissed, they are accumulating more and more adherents, and being treated with more and more respect.
EDIT: I exaggerate. EY has dealt with many aspects of FAI. But not, I think, with the most fundamental questions such as whether it makes any sense to talk about human values, whether preserving them is a good thing to do, how to trade off the present versus the future, and what “saving the human race” means.
Human preference is whatever you (being a human) happen to prefer, on reflection.
That is not a definition, no matter how many times it’s been repeated. It’s a tautology.
Sane definitions usually are. I don’t claim to know all about what sort of thing human preference is, but the term is defined roughly this way. This definition is itself fuzzy, because I can only refer to intuitions about “on reflection”, “prefer”, etc., but can’t define their combination in the concept of human preference mathematically. This definition contains an implicit problem statement, about formalization of the concept. But this formalization is the whole goal of preference theory, so one can’t expect it now. The term itself is useful, because it’s convenient to refer to the object of study.
FAI theory is an important topic not because it contains many interesting non-trivial results (it doesn’t), but because the problem needs to be solved. So far, even a good problem statement that won’t scare away mathematicians is lacking.
It’s an important topic, but I feel that may become an obstacle rather than a help towards the goal of avoiding AI catastrophe. It can be a flypaper that catches people interested in the problem, then leaves them stuck there while they wait for further clarifications from Eliezer that never come, instead of doing original work themselves, because they’ve been led to believe that FAI+CEV theory is more developed than it is.
I don’t think that was the intent, but it might be a welcome side-effect.
EY has little motivation to provide clarification, as long as people here continue to proclaim their faith in FAI+CEV. He’s said repeatedly that he doesn’t believe collaboration has value; he plans to solve the problem himself. Even supposing that he had a complete write-up on FAI+CEV in his hand today, actually publishing it could be a losing proposition in his eyes. It would encourage other people to do AI work and call it FAI (dangerous, I think he would say); it would make FAI no longer be the exclusive property of SIAI (a financial hazard); and it would reveal countless grounds for disagreement with his ideas and with his values.
Because I do believe in the value of collaboration, I would like to see more clarification. And I don’t think it’s forthcoming as long as people already give FAI+CEV the respect they would give a fully-formed theory.
Also, FAI+CEV is causing premature convergence within the transhumanist community. I know the standard FAI+CEV answers to a number of questions, and it dismays me to hear them spoken with more and more self-assurance by more and more smart people, when I know that these answers have weak spots that have been unexamined for far too long. It’s too soon for people to be agreeing this much on something that has been discussed so little.
Their mistake (I agree with your impression though). I’ve started working on FAI as soon as I understood the problem (as not having understanding of “fuzzy AGI” as a useful subgoal), about a year ago, and the current blog sequence is intended to help others in understanding the problem.
On the other hand, what do you see as the alternative to this “flypaper”, or an improvement thereof towards more productive modes? Building killer robots as a career is hardly a better road.
Gee, how can I answer this question in a way that doesn’t oblige me to do work?
One thing is, as a community, to motivate Eliezer to tell us more about his ideas on FAI on CEV, and to answer questions about them, by making it apparent that continuing to take these ideas seriously depends on continuing development of them. I very much appreciate his writing out his recent sequence on timeless decision theory, so I don’t want to harp on this at present. And of course Eliezer has no moral obligation to respond to you (unless you’ve given him time or money). But I’m not speaking of moral obligations; I’m speaking of strategy.
Another is to begin working on these ideas ourselves. This is hindered by us lacking a way to talk about, say, “Eliezer’s CEV” vs. CEV in general, and continuing to try to figure out what Eliezer’s opinion is (to get at the “true CEV theory”), instead of trying to figure out CEV theory independently. So a repeated pattern has been
person P (as in, for instance, “Phil”) asks a question about FAI or CEV
Eliezer doesn’t answer
person P gives their interpretation of FAI or CEV on the point, possibly in a “this is what I think Eliezer meant” way, or else in a “these are the implications of Eliezer’s ideas” way
Eliezer responds by saying that person P doesn’t know what they’re talking about, and should stop presuming to know what Eliezer thinks
What Steven said. If course, preference is not about preserving something we don’t want preserved, such as satisfaction of human drives as they currently are. Specifying the ways in which human values could grow is not vacuous, as some ways in which values could develop are better than others.
Back to definitions, human preference is whatever you (being a human) happen to prefer, on reflection. If you are right and stopping moral growth is undesirable (I agree), then by definition stopping moral growth is not part of human preference. And so on. Human preference is the specification at the top of meta, that describes all possible considerations about the ways in which all other relevant developments should happen.
I don’t think there is a top to meta; and if there is, there’s nothing human about it.
You are still speaking as if there were one privileged, appropriate level of analysis for values. In fact, as with everything expressed by human language, there are different levels of abstraction, that are appropriate in different circumstances.
The question of how meta to go, depends on the costs, the benefits, the certainty of the analysis, and other factors.
The question of how to go meta cannot be made independently of the very sorts of values that the friendly AI and CEV are themselves supposed to arbitrate between. There is no way to get outside the system and do it objectively.
That is not a definition, no matter how many times it’s been repeated. It’s a tautology. That is side-stepping the issue. You need to either start being specific about values, or stop asking people to respect Friendly AI and coherent extrapolated volition as if they were coherent ideas. I’ve been waiting years for an explanation, and yet these things are still developed only to the level of precision of a dope-fueled dormitory rap session. Yet, somehow, instead of being dismissed, they are accumulating more and more adherents, and being treated with more and more respect.
EDIT: I exaggerate. EY has dealt with many aspects of FAI. But not, I think, with the most fundamental questions such as whether it makes any sense to talk about human values, whether preserving them is a good thing to do, how to trade off the present versus the future, and what “saving the human race” means.
Sane definitions usually are. I don’t claim to know all about what sort of thing human preference is, but the term is defined roughly this way. This definition is itself fuzzy, because I can only refer to intuitions about “on reflection”, “prefer”, etc., but can’t define their combination in the concept of human preference mathematically. This definition contains an implicit problem statement, about formalization of the concept. But this formalization is the whole goal of preference theory, so one can’t expect it now. The term itself is useful, because it’s convenient to refer to the object of study.
FAI theory is an important topic not because it contains many interesting non-trivial results (it doesn’t), but because the problem needs to be solved. So far, even a good problem statement that won’t scare away mathematicians is lacking.
It’s an important topic, but I feel that may become an obstacle rather than a help towards the goal of avoiding AI catastrophe. It can be a flypaper that catches people interested in the problem, then leaves them stuck there while they wait for further clarifications from Eliezer that never come, instead of doing original work themselves, because they’ve been led to believe that FAI+CEV theory is more developed than it is.
I don’t think that was the intent, but it might be a welcome side-effect.
EY has little motivation to provide clarification, as long as people here continue to proclaim their faith in FAI+CEV. He’s said repeatedly that he doesn’t believe collaboration has value; he plans to solve the problem himself. Even supposing that he had a complete write-up on FAI+CEV in his hand today, actually publishing it could be a losing proposition in his eyes. It would encourage other people to do AI work and call it FAI (dangerous, I think he would say); it would make FAI no longer be the exclusive property of SIAI (a financial hazard); and it would reveal countless grounds for disagreement with his ideas and with his values.
Because I do believe in the value of collaboration, I would like to see more clarification. And I don’t think it’s forthcoming as long as people already give FAI+CEV the respect they would give a fully-formed theory.
Also, FAI+CEV is causing premature convergence within the transhumanist community. I know the standard FAI+CEV answers to a number of questions, and it dismays me to hear them spoken with more and more self-assurance by more and more smart people, when I know that these answers have weak spots that have been unexamined for far too long. It’s too soon for people to be agreeing this much on something that has been discussed so little.
Their mistake (I agree with your impression though). I’ve started working on FAI as soon as I understood the problem (as not having understanding of “fuzzy AGI” as a useful subgoal), about a year ago, and the current blog sequence is intended to help others in understanding the problem.
On the other hand, what do you see as the alternative to this “flypaper”, or an improvement thereof towards more productive modes? Building killer robots as a career is hardly a better road.
Gee, how can I answer this question in a way that doesn’t oblige me to do work?
One thing is, as a community, to motivate Eliezer to tell us more about his ideas on FAI on CEV, and to answer questions about them, by making it apparent that continuing to take these ideas seriously depends on continuing development of them. I very much appreciate his writing out his recent sequence on timeless decision theory, so I don’t want to harp on this at present. And of course Eliezer has no moral obligation to respond to you (unless you’ve given him time or money). But I’m not speaking of moral obligations; I’m speaking of strategy.
Another is to begin working on these ideas ourselves. This is hindered by us lacking a way to talk about, say, “Eliezer’s CEV” vs. CEV in general, and continuing to try to figure out what Eliezer’s opinion is (to get at the “true CEV theory”), instead of trying to figure out CEV theory independently. So a repeated pattern has been
person P (as in, for instance, “Phil”) asks a question about FAI or CEV
Eliezer doesn’t answer
person P gives their interpretation of FAI or CEV on the point, possibly in a “this is what I think Eliezer meant” way, or else in a “these are the implications of Eliezer’s ideas” way
Eliezer responds by saying that person P doesn’t know what they’re talking about, and should stop presuming to know what Eliezer thinks
end of discussion