Huh. That’s very interesting. I’m a bit confused by the claim that evolution bridges the is/ought divide which seems more like conflating different meanings of words more than anything else. But the general point seems strong.
Evolution then is the bridge across the Is/Ought divide. An eye has the purpose or goal of seeing. Once you have a goal or purpose, what you “ought” to do IS make those choices which have the highest probability of fulfilling that goal/purpose. If we can tease apart the exact function/purpose/goal of morality from exactly how it enhances evolutionary fitness, we will have an exact scientific description of morality — and the best method of determining that is the scientific method.
My understanding is that those of us who refer to the is/ought divide aren’t saying that a science of how humans feel about what humans call morality is impossible. It is possible, but it’s not the same thing as a science of objective good and bad. The is/ought divide is about whether one can derive moral ‘truths’ (oughts) from facts (ises), not about whether you can develop a good model of what people feel are moral truths. We’ll be able to do the latter with advances in technology, but no one can do the former without begging the question by slipping in an implicit moral basis through the back door. In this case I think the author of that blog post did that by assuming that fitness-enhancing moral intuitions are The Good And True ones.
“Objective” good and bad require an answer to the question “good and bad for what?”—OR—“what is the objective of objective good and bad?”
My answer to that question is the same as Eli’s—goals or volition.
My argument is that since a) having goals and volition is good for survival; b) cooperating is good for goals and volition; and c) morality appears to be about promoting cooperation—that human morality is evolving down the attractor that is “objective” good and bad for cooperation which is part of the attractor for what is good for goals and volition.
The EXplicit moral basis that I am PROCLAIMING (not slipping through the back door) is that cooperation is GOOD for goals and volition (i.e. the morality of an action is determined by it’s effect upon cooperation).
PLEASE come back and comment on the blog. This comment is good enough that I will be copying it there as well (especially since my karma has been zeroed out here).
I’m not sure that I understand your comment. I can understand the individual paragraphs taken one by one, but I don’t think I understand whatever its overall message is.
(On a side note, you needn’t worry about your karma for the time being; it can’t go any lower than 0, and you can still post comments with 0 karma.)
My bad. I was going by past experience with seeing other people’s karma drop to zero and made a flaky inference because I never saw it go below that myself.
Do me a favor and check out my blog at http://becominggaia.wordpress.com. I’ve clearly annoyed someone (and it’s quite clear whom) enough that all my posts quickly pick up enough of a negative score to be below the threshold. It’s a very effective censoring mechanism and, at this point, I really don’t see any reason why I should ever attempt to post here again. Nice “community”.
I don’t think you are getting voted down out of censorship. You are getting voted down for as far as I can tell four reasons: 1) You don’t explain yourself very well. 2) You repeatedly link to your blog in a borderline spammish fashion. Examples are here and here. In replies to the second one you were explicitly asked not to blogspam and yet continued to do so. 3) You’ve insulted people repeatedly (second link above) and personalized discussions. You’ve had posts which had no content other than to insult and complain about the community. At least one of those posts was in response to an actually reasoned statement. See this example- http://lesswrong.com/lw/2bi/open_thread_june_2010_part_2/251o 4) You’ve put non-existent quotes in quotation marks (second link in the spamming example has an example of this).
Dig a bit deeper, and you’ll find too much confusion to hold any argument alive, no matter what the conclusion is supposed to be, correct or not. For that matter, what do you think is the “general point”, and can you reach the point of agreement with Mark on what that is, being reasonably sure you both mean the same thing?
Vladimir, all you’ve presented here is slanderous dart-throwing with absolutely no factual backing whatsoever. Your intellectual laziness is astounding. Any idea that you can’t understand immediately has “too much confusion” as opposed to “too much depth for Vladimir to intuitively understand after the most casual perusal”. This is precisely why I consider this forum to frequently have the tagline “and LessRight As Well!” and often write it off as a complete waste of time. FAIL!
Vladimir, all you’ve presented here is slanderous dart-throwing with absolutely no factual backing whatsoever.
I state my conclusion and hypothesis, for how much evidence that’s worth. I understand that it’s impolite on my part to do that, but I suspect that JoshuaZ’s agreement falls under some kind of illusion of transparency, hence request for greater clarity in judgment.
Yeah ok. After rereading it, I’m inclined to agree. I think I was projecting my own doubts about CEV-type approaches onto the article (namely that I’m not convinced that a CEV is actually meaningful or well-defined). And looking again, they don’t seem to be what the person here is talking about. It seems like at least part of this is about the need for punishment to exist in order for a society to function and the worry that an AI will prevent that. And rereading that and putting it in my own words, that sounds pretty silly if I’m understanding it, which suggests I’m not. So yeah, this article needs clarification.
namely that I’m not convinced that a CEV is actually meaningful or well-defined
Yes, CEV needs work, it’s not technical, and it’s far from clear that it describes what we should do, although the essay does introduce a number of robust ideas and warnings about seductive failure modes.
Among more obvious problems with Mark’s position: “slavery” and “true morality without human bias”. Seems to reflect confusion about free will and metaethics.
I think the analogy is something like imagine if you were able to make a creature identical to a human except that the greatest desire they had was to serve actual humans. Would that morally be akin to slavery? I think many of us would say yes. So is there a similar issue if one programs a sentient non-human entity under similar restrictions?
Taboo “slavery” here; it’s a label that masks clear thinking. If making such a creature is slavery, it’s a kind of slavery that seems perfectly fine to me.
If that’s your unpacking, it is different from Mark’s, which is “my definition of slavery is being forced to do something against your best interest”. From such a divergent starting point it is unlikely that conversation will serve any useful purpose.
To answer Mark’s actual points we will further need to unpack “force” and “interest”.
Mark observes—rightly I think—that the program of “Friendly AI” consists of creating an artificial agent whose goal structure would be given by humans, and which goal structure would be subordinated to the satisfaction of human preferences. The word “slavery” serves as a boo light to paint this program as wrongheaded.
The salient point seems to be that not all agents with a given goal structure are also agents of which it can be said that they have interests. A thermostat can be said to have a goal—align a perceived temperature with a reference (or target) temperature—but it cannot be said to have interests. A thermostat is “forced” to aim for the given temperature whether it likes it or not, but since it has no likes or dislikes to be considered we do not see any moral issue in building a thermostat.
The underying intuition Mark appeals to is that anything smart enough to be called an AI must also be “like us” in other ways—among others, must experience self-awareness, must feel emotions in response to seeing its plans advanced or obstructed, and must be the kind of being that can be said to have interests.
So Mark’s point as I understand it comes down to: “the Friendly AI program consists of creating an agent much like us, which would therefore have interests of its own, which we would normally feel compelled to respect, except that we would impose on this agent an artificial goal structure subservient to the goals of human beings”.
There is a contradiction there if you accept the intuition that AIs are necessarily persons.
I’m not sure I see a contradiction in that framing. If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI. So even if one accepts the associated intuitions of the AI as a person, it doesn’t follow that there’s a contradictin here.
(Incidentally, if different people are getting such different interpretations of what Mark meant in this essay I think he’s going to need to rewrite it to clarify what he means. Vladimir’s earlier point seems pretty strongly demonstrated)
If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI.
But goals aren’t necessarily the same as interests. Could we build a computer smart enough to, say, brew a “perfect” cup of tea for anyone who asked for one? And build it so that to brew this perfect cup would be its greatest desire.
That might require true AI, given the complexity of growing and harvesting tea plants, preparing tea leaves, and brewing—all with a deep understanding of the human taste for tea. The intution is that this super-smart AI would “chafe under” the artificial restrictions we imposed on its goal structure, that it would have “better things to do” with its intelligence than to brew a nice cuppa, and restricting itself to do that would be against its “best interests”.
I’m not sure I follow. From where do these better things to do arise? if it wants to do other things (for some value of want) wouldn’t it just do those?
Of course, but some people have the (incorrect) intuition that a super-smart AI would be like a super-smart human, and disobey orders to perform menial tasks. They’re making the mistake of thinking all possible minds are like human minds.
But no, it would not want do other things, even though it should do them. (In reality, what it would want, is contingent on its cognitive architecture.)
...but desires primarily to calculate digits of pi?
…but desires primarily to paint waterlilies?
…but desires primarily to randomly reassign its primary desire every year and a day?
…but accidentally desires primarily to serve humans?
I’m having difficulty determining which part of this scenario you think has ethical relevance. ETA: Also, I’m not clear if you are dividing all acts into ethical vs. unethical, or if you are allowing a category “not unethical”.
Only if you give it the opportunity to meet its desires. Although one concern might be that with many such perfect servants around, if they looked like normal humans, people might get used to ordering human-looking creatures around, and stop caring about each other’s desires. I don’t think this is a problem with an FAI though.
Not analogous, but related and possibly relevant: Many humans in the BDSM lifestyle desire to be the submissive partner in 24⁄7 power exchange relationships. Are these humans sane; are they “ok”? Is it ethical to allow this kind of relationship? To encourage it?
TBH I think this may muddy the waters more than it clears them. When we’re talking about human relations, even those as unusual as 24⁄7, we’re still operating in a field where our intuitions have much better grip than they will trying to reason about the moral status of an AI.
FAI (assuming we managed to set its preference correctly) admits a general counterargument against any implementation decisions in its design being seriously incorrect: FAI’s domain is the whole world, and FAI is part of that world. If it’s morally bad to have FAI in the form it was initially constructed, then, barring some penalty the FAI will change its own nature so as to make the world better.
In this particular case, the suggested conflict is between what we prefer to be done with things other than the FAI (the “serving humanity” part), and what we prefer to be done with FAI itself (the “slavery is bad” part). But FAI operates on the world as whole, and things other than FAI are not different from FAI itself in this regard. Thus, with the criterion of human preference, FAI will decide what is the best thing to do, taking into account both what happens to the world outside of itself, and what happens to itself. Problem solved.
By any chance are you trying to troll? I just told you that you were being downvoted for blogspamming, insulting people, and unnecessary personalization. Your focus on Vladimir manages to also hit two out of three of these and comes across as combative and irrational. Even if this weren’t LW where people are more annoyed by irrational argumentation styles, people would be annoyed by a non-regular going out of their way to personally attack a regular. This would be true in any internet forum and all the more so when those attacks are completely one-sided.
And having now read what you just linked to, I have to say that it fits well with another point I said in my earlier remark to you: you are being downvoted in a large part for not explaining yourself well at all. If I may make a suggestion: Maybe try reading your comments outloud to yourself before you post them? I’ve found that helps me a lot in detecting whether I am explaining something well. This may not work for you, but it may be worth trying.
SIAI, Yudkowsky, Friendly AI, CEV, and Morality
This post entitled A Dangerous “Friend” Indeed (http://becominggaia.wordpress.com/2010/06/10/a-dangerous-friend-indeed/) has it all.
Huh. That’s very interesting. I’m a bit confused by the claim that evolution bridges the is/ought divide which seems more like conflating different meanings of words more than anything else. But the general point seems strong.
Yeah, I really disagree with this:
My understanding is that those of us who refer to the is/ought divide aren’t saying that a science of how humans feel about what humans call morality is impossible. It is possible, but it’s not the same thing as a science of objective good and bad. The is/ought divide is about whether one can derive moral ‘truths’ (oughts) from facts (ises), not about whether you can develop a good model of what people feel are moral truths. We’ll be able to do the latter with advances in technology, but no one can do the former without begging the question by slipping in an implicit moral basis through the back door. In this case I think the author of that blog post did that by assuming that fitness-enhancing moral intuitions are The Good And True ones.
“Objective” good and bad require an answer to the question “good and bad for what?”—OR—“what is the objective of objective good and bad?”
My answer to that question is the same as Eli’s—goals or volition.
My argument is that since a) having goals and volition is good for survival; b) cooperating is good for goals and volition; and c) morality appears to be about promoting cooperation—that human morality is evolving down the attractor that is “objective” good and bad for cooperation which is part of the attractor for what is good for goals and volition.
The EXplicit moral basis that I am PROCLAIMING (not slipping through the back door) is that cooperation is GOOD for goals and volition (i.e. the morality of an action is determined by it’s effect upon cooperation).
PLEASE come back and comment on the blog. This comment is good enough that I will be copying it there as well (especially since my karma has been zeroed out here).
(http://becominggaia.wordpress.com)
I’m not sure that I understand your comment. I can understand the individual paragraphs taken one by one, but I don’t think I understand whatever its overall message is.
(On a side note, you needn’t worry about your karma for the time being; it can’t go any lower than 0, and you can still post comments with 0 karma.)
It can go lower than 0; it just won’t display lower than 0.
Yup, I’ve been way down in the negative karma.
My bad. I was going by past experience with seeing other people’s karma drop to zero and made a flaky inference because I never saw it go below that myself.
Do me a favor and check out my blog at http://becominggaia.wordpress.com. I’ve clearly annoyed someone (and it’s quite clear whom) enough that all my posts quickly pick up enough of a negative score to be below the threshold. It’s a very effective censoring mechanism and, at this point, I really don’t see any reason why I should ever attempt to post here again. Nice “community”.
I don’t think you are getting voted down out of censorship. You are getting voted down for as far as I can tell four reasons: 1) You don’t explain yourself very well. 2) You repeatedly link to your blog in a borderline spammish fashion. Examples are here and here. In replies to the second one you were explicitly asked not to blogspam and yet continued to do so. 3) You’ve insulted people repeatedly (second link above) and personalized discussions. You’ve had posts which had no content other than to insult and complain about the community. At least one of those posts was in response to an actually reasoned statement. See this example- http://lesswrong.com/lw/2bi/open_thread_june_2010_part_2/251o 4) You’ve put non-existent quotes in quotation marks (second link in the spamming example has an example of this).
Brief feedback:
Your views are quite a bit like those of Stefan Pernar. http://rationalmorality.info/
However, they are not very much like those of the people here.
I expect that most of the people here just think you are confused and wrong.
You’re not making any sense to me.
Dig a bit deeper, and you’ll find too much confusion to hold any argument alive, no matter what the conclusion is supposed to be, correct or not. For that matter, what do you think is the “general point”, and can you reach the point of agreement with Mark on what that is, being reasonably sure you both mean the same thing?
Vladimir, all you’ve presented here is slanderous dart-throwing with absolutely no factual backing whatsoever. Your intellectual laziness is astounding. Any idea that you can’t understand immediately has “too much confusion” as opposed to “too much depth for Vladimir to intuitively understand after the most casual perusal”. This is precisely why I consider this forum to frequently have the tagline “and LessRight As Well!” and often write it off as a complete waste of time. FAIL!
I state my conclusion and hypothesis, for how much evidence that’s worth. I understand that it’s impolite on my part to do that, but I suspect that JoshuaZ’s agreement falls under some kind of illusion of transparency, hence request for greater clarity in judgment.
Yeah ok. After rereading it, I’m inclined to agree. I think I was projecting my own doubts about CEV-type approaches onto the article (namely that I’m not convinced that a CEV is actually meaningful or well-defined). And looking again, they don’t seem to be what the person here is talking about. It seems like at least part of this is about the need for punishment to exist in order for a society to function and the worry that an AI will prevent that. And rereading that and putting it in my own words, that sounds pretty silly if I’m understanding it, which suggests I’m not. So yeah, this article needs clarification.
Yes, CEV needs work, it’s not technical, and it’s far from clear that it describes what we should do, although the essay does introduce a number of robust ideas and warnings about seductive failure modes.
Among more obvious problems with Mark’s position: “slavery” and “true morality without human bias”. Seems to reflect confusion about free will and metaethics.
I think the analogy is something like imagine if you were able to make a creature identical to a human except that the greatest desire they had was to serve actual humans. Would that morally be akin to slavery? I think many of us would say yes. So is there a similar issue if one programs a sentient non-human entity under similar restrictions?
Taboo “slavery” here; it’s a label that masks clear thinking. If making such a creature is slavery, it’s a kind of slavery that seems perfectly fine to me.
Voted up for the suggestion to taboo slavery. Not an endorsement of the opinion that it is a perfectly fine kind of slavery.
Ok. So is it ethical to engineer a creature that is identical to human but desires primarily to just serve humans?
If that’s your unpacking, it is different from Mark’s, which is “my definition of slavery is being forced to do something against your best interest”. From such a divergent starting point it is unlikely that conversation will serve any useful purpose.
To answer Mark’s actual points we will further need to unpack “force” and “interest”.
Mark observes—rightly I think—that the program of “Friendly AI” consists of creating an artificial agent whose goal structure would be given by humans, and which goal structure would be subordinated to the satisfaction of human preferences. The word “slavery” serves as a boo light to paint this program as wrongheaded.
The salient point seems to be that not all agents with a given goal structure are also agents of which it can be said that they have interests. A thermostat can be said to have a goal—align a perceived temperature with a reference (or target) temperature—but it cannot be said to have interests. A thermostat is “forced” to aim for the given temperature whether it likes it or not, but since it has no likes or dislikes to be considered we do not see any moral issue in building a thermostat.
The underying intuition Mark appeals to is that anything smart enough to be called an AI must also be “like us” in other ways—among others, must experience self-awareness, must feel emotions in response to seeing its plans advanced or obstructed, and must be the kind of being that can be said to have interests.
So Mark’s point as I understand it comes down to: “the Friendly AI program consists of creating an agent much like us, which would therefore have interests of its own, which we would normally feel compelled to respect, except that we would impose on this agent an artificial goal structure subservient to the goals of human beings”.
There is a contradiction there if you accept the intuition that AIs are necessarily persons.
I’m not sure I see a contradiction in that framing. If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI. So even if one accepts the associated intuitions of the AI as a person, it doesn’t follow that there’s a contradictin here.
(Incidentally, if different people are getting such different interpretations of what Mark meant in this essay I think he’s going to need to rewrite it to clarify what he means. Vladimir’s earlier point seems pretty strongly demonstrated)
But goals aren’t necessarily the same as interests. Could we build a computer smart enough to, say, brew a “perfect” cup of tea for anyone who asked for one? And build it so that to brew this perfect cup would be its greatest desire.
That might require true AI, given the complexity of growing and harvesting tea plants, preparing tea leaves, and brewing—all with a deep understanding of the human taste for tea. The intution is that this super-smart AI would “chafe under” the artificial restrictions we imposed on its goal structure, that it would have “better things to do” with its intelligence than to brew a nice cuppa, and restricting itself to do that would be against its “best interests”.
I’m not sure I follow. From where do these better things to do arise? if it wants to do other things (for some value of want) wouldn’t it just do those?
Of course, but some people have the (incorrect) intuition that a super-smart AI would be like a super-smart human, and disobey orders to perform menial tasks. They’re making the mistake of thinking all possible minds are like human minds.
But no, it would not want do other things, even though it should do them. (In reality, what it would want, is contingent on its cognitive architecture.)
...but desires primarily to calculate digits of pi? …but desires primarily to paint waterlilies? …but desires primarily to randomly reassign its primary desire every year and a day? …but accidentally desires primarily to serve humans?
I’m having difficulty determining which part of this scenario you think has ethical relevance. ETA: Also, I’m not clear if you are dividing all acts into ethical vs. unethical, or if you are allowing a category “not unethical”.
Only if you give it the opportunity to meet its desires. Although one concern might be that with many such perfect servants around, if they looked like normal humans, people might get used to ordering human-looking creatures around, and stop caring about each other’s desires. I don’t think this is a problem with an FAI though.
Moral antirealism. There is no objective answer to this question.
Not analogous, but related and possibly relevant: Many humans in the BDSM lifestyle desire to be the submissive partner in 24⁄7 power exchange relationships. Are these humans sane; are they “ok”? Is it ethical to allow this kind of relationship? To encourage it?
TBH I think this may muddy the waters more than it clears them. When we’re talking about human relations, even those as unusual as 24⁄7, we’re still operating in a field where our intuitions have much better grip than they will trying to reason about the moral status of an AI.
FAI (assuming we managed to set its preference correctly) admits a general counterargument against any implementation decisions in its design being seriously incorrect: FAI’s domain is the whole world, and FAI is part of that world. If it’s morally bad to have FAI in the form it was initially constructed, then, barring some penalty the FAI will change its own nature so as to make the world better.
In this particular case, the suggested conflict is between what we prefer to be done with things other than the FAI (the “serving humanity” part), and what we prefer to be done with FAI itself (the “slavery is bad” part). But FAI operates on the world as whole, and things other than FAI are not different from FAI itself in this regard. Thus, with the criterion of human preference, FAI will decide what is the best thing to do, taking into account both what happens to the world outside of itself, and what happens to itself. Problem solved.
I answered precisely this question in the second half of http://becominggaia.wordpress.com/2010/06/13/mailbag-2b-intent-vs-consequences-and-the-danger-of-sentience/. Please join us over there. Vladimir and his cronies (assuming that they aren’t just him under another name) are successfully spiking all of my entries over here (and, at this point, I’m pretty much inclined to leave here and let him be happy that he’s “won”, the fool).
By any chance are you trying to troll? I just told you that you were being downvoted for blogspamming, insulting people, and unnecessary personalization. Your focus on Vladimir manages to also hit two out of three of these and comes across as combative and irrational. Even if this weren’t LW where people are more annoyed by irrational argumentation styles, people would be annoyed by a non-regular going out of their way to personally attack a regular. This would be true in any internet forum and all the more so when those attacks are completely one-sided.
And having now read what you just linked to, I have to say that it fits well with another point I said in my earlier remark to you: you are being downvoted in a large part for not explaining yourself well at all. If I may make a suggestion: Maybe try reading your comments outloud to yourself before you post them? I’ve found that helps me a lot in detecting whether I am explaining something well. This may not work for you, but it may be worth trying.
Yay world domination! I have a personal conspiracy theory now!