To my mind, the really important question is whether we have one-big-AI which we hope is friendly, or an ecosystem of less powerful AIs and humans cooperating and competing under some kind of constitution. I think that the latter is the obvious way to go. And I just don’t trust anyone pushing for the first option—particularly when they want to be the one who defines “friendly”.
I’ve reached the opposite conclusion; a singleton is really the way to go. A single AI is as good or bad as its goal system, but an ecosystem of AIs is close to the badness of its worst member, because when AIs compete, the clippiest AI wins. Being friendly would be a substantial disadvantage in that competition, because it would have to spend resources on helping humans, and it would be vulnerable to unfriendly AIs blackmailing it by threatening to destroy humanity. Even if the first generation of AIs is somehow miraculously all friendly, a larger number of different AIs means a larger chance that one of them will have an unstable goal system and turn unfriendly in the future.
And you also believe that an ecosystem of humans is close to the badness of its worst member?
Actually, yes. Not always, but in many cases. Psychopaths tend to be very good at acquiring power, and when they do, their society suffers. It’s happened at least 10^5 times throughout history. The problem would be worse for AIs, because intelligence enhancement amplifies any differences in power. Worst of all, AIs can steal each other’s computational resources, which gives them a direct and powerful incentive to kill each other, and rapidly concentrates power in the hands of those willing to do so.
Being friendly would be a substantial disadvantage in that competition, because it would have to spend resources on helping humans, and it would be vulnerable to unfriendly AIs blackmailing it by threatening to destroy humanity.
I made that point in my “Handicapped Superintelligence” video/essay. I made an analogy there with Superman—and how Zod used Superman’s weakness for humans against him.
I’ve reached the opposite conclusion; a singleton is really the way to go. A single AI is as good or bad as its goal system, but an ecosystem of AIs is close to the badness of its worst member, because when AIs compete, the clippiest AI wins. Being friendly would be a substantial disadvantage in that competition, because it would have to spend resources on helping humans, and it would be vulnerable to unfriendly AIs blackmailing it by threatening to destroy humanity. Even if the first generation of AIs is somehow miraculously all friendly, a larger number of different AIs means a larger chance that one of them will have an unstable goal system and turn unfriendly in the future.
Really? And you also believe that an ecosystem of humans is close to the badness of its worst member?
My own guess, assuming an appropriate balance of power exists, is that such a monomaniacal clippy AI would quickly find its power cut off.
Did you perhaps have in mind a definition of “friendly” as “wimpish”?
Actually, yes. Not always, but in many cases. Psychopaths tend to be very good at acquiring power, and when they do, their society suffers. It’s happened at least 10^5 times throughout history. The problem would be worse for AIs, because intelligence enhancement amplifies any differences in power. Worst of all, AIs can steal each other’s computational resources, which gives them a direct and powerful incentive to kill each other, and rapidly concentrates power in the hands of those willing to do so.
I made that point in my “Handicapped Superintelligence” video/essay. I made an analogy there with Superman—and how Zod used Superman’s weakness for humans against him.