Ah. I interpreted the challenge “Oh, so you get to say what ‘Friendly’ means” to mean that the speaker is objecting that the listener seems to feel that he’s entitled to say what the AI is supposed to do. You seem to be having a different question of what’s the intended general meaning of the term.
I had taken as given that a “Friendly” AI is, by definition, one that you trust to do what you want, even if it’s much smarter and more powerful than you. If your desires are contradictory, it should still do something reasonable.
This leaves “reasonable” and “what you want” undefined. Filling in those definitions with something technically well-defined and getting to the point where you can reasonably expect it to be stable when it’s much smarter than it’s designers is the crux of the problem.
Wikipedia’s definition at http://en.wikipedia.org/wiki/Friendly_artificial_intelligence is: “A Friendly Artificial Intelligence or FAI is an artificial intelligence (AI) that has a positive rather than negative effect on humanity.” Of course, this leaves “positive” undefined, which is not any better than leaving “what you want” or “reasonable” undefined.
At the end of the day, you don’t need the concept. The question the creators of the AI will ask themselves will be “Do we want to run this AI?”
I agree that that’s a valid worry, but I intended to look at things from the point of view of the people making the AI so I was expressing a different worry. I edited my post to clarify.
Ah. I interpreted the challenge “Oh, so you get to say what ‘Friendly’ means” to mean that the speaker is objecting that the listener seems to feel that he’s entitled to say what the AI is supposed to do. You seem to be having a different question of what’s the intended general meaning of the term.
I had taken as given that a “Friendly” AI is, by definition, one that you trust to do what you want, even if it’s much smarter and more powerful than you. If your desires are contradictory, it should still do something reasonable.
This leaves “reasonable” and “what you want” undefined. Filling in those definitions with something technically well-defined and getting to the point where you can reasonably expect it to be stable when it’s much smarter than it’s designers is the crux of the problem.
Wikipedia’s definition at http://en.wikipedia.org/wiki/Friendly_artificial_intelligence is: “A Friendly Artificial Intelligence or FAI is an artificial intelligence (AI) that has a positive rather than negative effect on humanity.” Of course, this leaves “positive” undefined, which is not any better than leaving “what you want” or “reasonable” undefined.
At the end of the day, you don’t need the concept. The question the creators of the AI will ask themselves will be “Do we want to run this AI?”
I worry that the question will be “What do you mean we, Tonto?”
I agree that that’s a valid worry, but I intended to look at things from the point of view of the people making the AI so I was expressing a different worry. I edited my post to clarify.