I’m not sure I see a contradiction in that framing. If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI. So even if one accepts the associated intuitions of the AI as a person, it doesn’t follow that there’s a contradictin here.
(Incidentally, if different people are getting such different interpretations of what Mark meant in this essay I think he’s going to need to rewrite it to clarify what he means. Vladimir’s earlier point seems pretty strongly demonstrated)
If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI.
But goals aren’t necessarily the same as interests. Could we build a computer smart enough to, say, brew a “perfect” cup of tea for anyone who asked for one? And build it so that to brew this perfect cup would be its greatest desire.
That might require true AI, given the complexity of growing and harvesting tea plants, preparing tea leaves, and brewing—all with a deep understanding of the human taste for tea. The intution is that this super-smart AI would “chafe under” the artificial restrictions we imposed on its goal structure, that it would have “better things to do” with its intelligence than to brew a nice cuppa, and restricting itself to do that would be against its “best interests”.
I’m not sure I follow. From where do these better things to do arise? if it wants to do other things (for some value of want) wouldn’t it just do those?
Of course, but some people have the (incorrect) intuition that a super-smart AI would be like a super-smart human, and disobey orders to perform menial tasks. They’re making the mistake of thinking all possible minds are like human minds.
But no, it would not want do other things, even though it should do them. (In reality, what it would want, is contingent on its cognitive architecture.)
I’m not sure I see a contradiction in that framing. If we’ve programmed the AI then its interests precisely align with ours if it really is an FAI. So even if one accepts the associated intuitions of the AI as a person, it doesn’t follow that there’s a contradictin here.
(Incidentally, if different people are getting such different interpretations of what Mark meant in this essay I think he’s going to need to rewrite it to clarify what he means. Vladimir’s earlier point seems pretty strongly demonstrated)
But goals aren’t necessarily the same as interests. Could we build a computer smart enough to, say, brew a “perfect” cup of tea for anyone who asked for one? And build it so that to brew this perfect cup would be its greatest desire.
That might require true AI, given the complexity of growing and harvesting tea plants, preparing tea leaves, and brewing—all with a deep understanding of the human taste for tea. The intution is that this super-smart AI would “chafe under” the artificial restrictions we imposed on its goal structure, that it would have “better things to do” with its intelligence than to brew a nice cuppa, and restricting itself to do that would be against its “best interests”.
I’m not sure I follow. From where do these better things to do arise? if it wants to do other things (for some value of want) wouldn’t it just do those?
Of course, but some people have the (incorrect) intuition that a super-smart AI would be like a super-smart human, and disobey orders to perform menial tasks. They’re making the mistake of thinking all possible minds are like human minds.
But no, it would not want do other things, even though it should do them. (In reality, what it would want, is contingent on its cognitive architecture.)