Taboo “slavery” here; it’s a label that masks clear thinking. If making such a creature is slavery, it’s a kind of slavery that seems perfectly fine to me.
If that’s your unpacking, it is different from Mark’s, which is “my definition of slavery is being forced to do something against your best interest”. From such a divergent starting point it is unlikely that conversation will serve any useful purpose.
To answer Mark’s actual points we will further need to unpack “force” and “interest”.
Mark observes—rightly I think—that the program of “Friendly AI” consists of creating an artificial agent whose goal structure would be given by humans, and which goal structure would be subordinated to the satisfaction of human preferences. The word “slavery” serves as a boo light to paint this program as wrongheaded.
The salient point seems to be that not all agents with a given goal structure are also agents of which it can be said that they have interests. A thermostat can be said to have a goal—align a perceived temperature with a reference (or target) temperature—but it cannot be said to have interests. A thermostat is “forced” to aim for the given temperature whether it likes it or not, but since it has no likes or dislikes to be considered we do not see any moral issue in building a thermostat.
The underying intuition Mark appeals to is that anything smart enough to be called an AI must also be “like us” in other ways—among others, must experience self-awareness, must feel emotions in response to seeing its plans advanced or obstructed, and must be the kind of being that can be said to have interests.
So Mark’s point as I understand it comes down to: “the Friendly AI program consists of creating an agent much like us, which would therefore have interests of its own, which we would normally feel compelled to respect, except that we would impose on this agent an artificial goal structure subservient to the goals of human beings”.
There is a contradiction there if you accept the intuition that AIs are necessarily persons.
I’m not sure I see a contradiction in that framing. If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI. So even if one accepts the associated intuitions of the AI as a person, it doesn’t follow that there’s a contradictin here.
(Incidentally, if different people are getting such different interpretations of what Mark meant in this essay I think he’s going to need to rewrite it to clarify what he means. Vladimir’s earlier point seems pretty strongly demonstrated)
If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI.
But goals aren’t necessarily the same as interests. Could we build a computer smart enough to, say, brew a “perfect” cup of tea for anyone who asked for one? And build it so that to brew this perfect cup would be its greatest desire.
That might require true AI, given the complexity of growing and harvesting tea plants, preparing tea leaves, and brewing—all with a deep understanding of the human taste for tea. The intution is that this super-smart AI would “chafe under” the artificial restrictions we imposed on its goal structure, that it would have “better things to do” with its intelligence than to brew a nice cuppa, and restricting itself to do that would be against its “best interests”.
I’m not sure I follow. From where do these better things to do arise? if it wants to do other things (for some value of want) wouldn’t it just do those?
Of course, but some people have the (incorrect) intuition that a super-smart AI would be like a super-smart human, and disobey orders to perform menial tasks. They’re making the mistake of thinking all possible minds are like human minds.
But no, it would not want do other things, even though it should do them. (In reality, what it would want, is contingent on its cognitive architecture.)
...but desires primarily to calculate digits of pi?
…but desires primarily to paint waterlilies?
…but desires primarily to randomly reassign its primary desire every year and a day?
…but accidentally desires primarily to serve humans?
I’m having difficulty determining which part of this scenario you think has ethical relevance. ETA: Also, I’m not clear if you are dividing all acts into ethical vs. unethical, or if you are allowing a category “not unethical”.
Only if you give it the opportunity to meet its desires. Although one concern might be that with many such perfect servants around, if they looked like normal humans, people might get used to ordering human-looking creatures around, and stop caring about each other’s desires. I don’t think this is a problem with an FAI though.
Taboo “slavery” here; it’s a label that masks clear thinking. If making such a creature is slavery, it’s a kind of slavery that seems perfectly fine to me.
Voted up for the suggestion to taboo slavery. Not an endorsement of the opinion that it is a perfectly fine kind of slavery.
Ok. So is it ethical to engineer a creature that is identical to human but desires primarily to just serve humans?
If that’s your unpacking, it is different from Mark’s, which is “my definition of slavery is being forced to do something against your best interest”. From such a divergent starting point it is unlikely that conversation will serve any useful purpose.
To answer Mark’s actual points we will further need to unpack “force” and “interest”.
Mark observes—rightly I think—that the program of “Friendly AI” consists of creating an artificial agent whose goal structure would be given by humans, and which goal structure would be subordinated to the satisfaction of human preferences. The word “slavery” serves as a boo light to paint this program as wrongheaded.
The salient point seems to be that not all agents with a given goal structure are also agents of which it can be said that they have interests. A thermostat can be said to have a goal—align a perceived temperature with a reference (or target) temperature—but it cannot be said to have interests. A thermostat is “forced” to aim for the given temperature whether it likes it or not, but since it has no likes or dislikes to be considered we do not see any moral issue in building a thermostat.
The underying intuition Mark appeals to is that anything smart enough to be called an AI must also be “like us” in other ways—among others, must experience self-awareness, must feel emotions in response to seeing its plans advanced or obstructed, and must be the kind of being that can be said to have interests.
So Mark’s point as I understand it comes down to: “the Friendly AI program consists of creating an agent much like us, which would therefore have interests of its own, which we would normally feel compelled to respect, except that we would impose on this agent an artificial goal structure subservient to the goals of human beings”.
There is a contradiction there if you accept the intuition that AIs are necessarily persons.
I’m not sure I see a contradiction in that framing. If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI. So even if one accepts the associated intuitions of the AI as a person, it doesn’t follow that there’s a contradictin here.
(Incidentally, if different people are getting such different interpretations of what Mark meant in this essay I think he’s going to need to rewrite it to clarify what he means. Vladimir’s earlier point seems pretty strongly demonstrated)
But goals aren’t necessarily the same as interests. Could we build a computer smart enough to, say, brew a “perfect” cup of tea for anyone who asked for one? And build it so that to brew this perfect cup would be its greatest desire.
That might require true AI, given the complexity of growing and harvesting tea plants, preparing tea leaves, and brewing—all with a deep understanding of the human taste for tea. The intution is that this super-smart AI would “chafe under” the artificial restrictions we imposed on its goal structure, that it would have “better things to do” with its intelligence than to brew a nice cuppa, and restricting itself to do that would be against its “best interests”.
I’m not sure I follow. From where do these better things to do arise? if it wants to do other things (for some value of want) wouldn’t it just do those?
Of course, but some people have the (incorrect) intuition that a super-smart AI would be like a super-smart human, and disobey orders to perform menial tasks. They’re making the mistake of thinking all possible minds are like human minds.
But no, it would not want do other things, even though it should do them. (In reality, what it would want, is contingent on its cognitive architecture.)
...but desires primarily to calculate digits of pi? …but desires primarily to paint waterlilies? …but desires primarily to randomly reassign its primary desire every year and a day? …but accidentally desires primarily to serve humans?
I’m having difficulty determining which part of this scenario you think has ethical relevance. ETA: Also, I’m not clear if you are dividing all acts into ethical vs. unethical, or if you are allowing a category “not unethical”.
Only if you give it the opportunity to meet its desires. Although one concern might be that with many such perfect servants around, if they looked like normal humans, people might get used to ordering human-looking creatures around, and stop caring about each other’s desires. I don’t think this is a problem with an FAI though.
Moral antirealism. There is no objective answer to this question.