That’s the plot of the Terminator movies, but it doesn’t seem to be a likely scenario.
During their regime, the Nazis locked up, used as slave labor, and eventually killed, millions of people. Most of them were Ashkenazi Jews, perhaps the smartest of all ethnic groups, with a language difficult to comprehend to outsiders, living in close-knit communities with transnational range, and strong inter-community ties. Did they get “out of the box” and take over the Third Reich? Nope.
AIs might have some advantages for being digital, but also disadvantages.
I think you miss the part where the team of millions continues its self-copying until it eats up every available computing power. If there’s any significant computing overhang, the AI could easily seize control of way more computing power than all the human brains put together.
Also, I think you underestimate the “highly coordinated” part. Any copy of the AI will likely share the exact same goals, and the exact same beliefs. Its instances will have common knowledge of this fact. This would creates an unprecedented level of trust. (The only possible exception I can think of are twins. And even so…)
So, let’s recap:
Thinks 100 times faster than a human, though no better.
Can copy itself over many times (the exact amount depends on computing power available).
The resulting team forms a nearly perfectly coordinated group.
Do you at least concede that this is potentially more dangerous than a whole country armed up with nukes? Would you rely on it being less dangerous than that?
When I imagine that I could make my copy which would be identical to me, sharing my goals, able to copy its experiences back to me, and willing to die for me (something like Naruto’s clones), taking over the society seems rather easy. (Assuming that no one else has this ability, and no one suspects me of having it. In real life it would probably help if all the clones looked different, but had an ability to recognize each other.)
Research: For each interesting topic I could make dozen clones which would study the topic in libraries and universities, and discuss their findings with each other. I don’t suppose it would make me an expert on everything, but I could get at least all the university-level education on most things. Resources: If I can make more money than I spend, and if I don’t make too much copies to imbalance the economy, I can let a few dozen clones work and produce the money for the rest of them. At least in the starting phase, until my research groups discover better ways to make money. Contacts: Different clones could go to different places making useful contacts wil different kinds of people. Sometimes you find a person which can help your goals significantly. With many clones I could make contacts in many different social groups, and overcome language or religious barriers (I can have a few clones learn the language or join the religion). Multiple “first impressions”: If I need a help of a given person or organization, I could in many cases gain their trust by sending multiple different clones to them, using different strategies to befriend them, until I find one that works. Taking over democratic organizations: Any organization with low barriers to entry and democratic voting can be taken over by sending enough clones there, and then voting some of the clones as new leaders. A typical non-governmental organization or even a smaller political party could be gained this way. I don’t even need a majority of clones there: two potential leaders competing with each other, half dozen experts openly supporting each of them, and dozen people befriending random voters and explaining them why leader X or leader Y is the perfect choice; then most of the voting would be done by other people. Assassination: If someone is too much of a problem, I can create a clone which kills them and then disappears. This should be used very rarely, not to draw attention to my abilities. Safety: To protect myself, I would send my different clones to different countries over the world. Joining all the winning sides: If there is an important group of people, I could join them, even the groups fighting against each other. Whoever wins, some of my clones are on the winning side.
If that AI runs on expensive or specialized hardware, it can’t necessarily expand much. For instance, if it runs on hardware worth millions of dollars, it can’t exactly copy itself just anywhere yet. Assuming that the first AI of that level will be cutting edge research and won’t be cheap, that gives a certain time window to study it safely.
The AI may be dangerous if it appeared now, but if it appears in, say, fifty years, then it will have to deal with the state of the art fifty years from now. Expanding without getting caught might be considerably more difficult then than it is now—weak AI will be all over the place, for one.
Last, but not least, the AI must have access to its own source code in order to copy it. That’s far from a given, especially if it’s a neural architecture. A human-level AI would not know how it works any more than we know how we work, so if it has no read access to itself or no way to probe its own circuitry, it won’t be able to copy itself at all. I doubt the first AI would actually have fine-grained access to its own inner workings, and I doubt it would have anywhere close to the amount of resources required to reverse engineer itself. Of course, that point is moot if some fool does give it access...
I agree with your first point, though it gets worse for us as hardware gets cheaper and cheaper.
I like your second point even more: it’s actionable. We could work on the security of personal computers.
That last one is incorrect however. The AI only have to access its object code in order to copy itself. That’s something even current computer viruses can do. And we’re back to boxing it.
If the AI is a learning system such as a neural network, and I believe that’s quite likely to be the case, there is no source/object dichotomy at all and the code may very well be unreadable outside of simple local update procedures that are completely out of the AI’s control. In other words, it might be physically impossible for both the AI and ourselves to access the AI’s object code—it would be locked in a hardware box with no physical wires to probe its contents, basically.
I mean, think of a physical hardware circuit implementing a kind of neuron network—in order for the network to be “copiable”, you need to be able to read the values of all neurons. However, that requires a global clock (to ensure synchronization, though AI might tolerate being a bit out of phase) and a large number of extra wires connecting each component to busses going out of the system. Of course, all that extra fluff inflates the cost of the system, makes it bigger, slower and probably less energy efficient. Since the first human-level AI won’t just come out of nowhere, it will probably use off-the-shelf digital neural components, and for cost and speed reasons, these components might not actually offer any way to copy their contents.
This being said, even if the AI runs on conventional hardware, locking it out of its own object code isn’t exactly rocket science. The specification of some programming languages already guarantee that this cannot happen, and type/proof theory is an active research field that may very well be able to prove the conformance of implementation to specification. If the AI is a neural network emulated on conventional hardware, the risks that it can read itself without permission are basically zilch.
There are various notions of intelligence, social intelligence includes the skills for getting in charge. My point is that human-level intelligence, even replicated or sped up, is generally not enough.
That’s the plot of the Terminator movies, but it doesn’t seem to be a likely scenario.
During their regime, the Nazis locked up, used as slave labor, and eventually killed, millions of people. Most of them were Ashkenazi Jews, perhaps the smartest of all ethnic groups, with a language difficult to comprehend to outsiders, living in close-knit communities with transnational range, and strong inter-community ties.
Did they get “out of the box” and take over the Third Reich? Nope.
AIs might have some advantages for being digital, but also disadvantages.
I think you miss the part where the team of millions continues its self-copying until it eats up every available computing power. If there’s any significant computing overhang, the AI could easily seize control of way more computing power than all the human brains put together.
Also, I think you underestimate the “highly coordinated” part. Any copy of the AI will likely share the exact same goals, and the exact same beliefs. Its instances will have common knowledge of this fact. This would creates an unprecedented level of trust. (The only possible exception I can think of are twins. And even so…)
So, let’s recap:
Thinks 100 times faster than a human, though no better.
Can copy itself over many times (the exact amount depends on computing power available).
The resulting team forms a nearly perfectly coordinated group.
Do you at least concede that this is potentially more dangerous than a whole country armed up with nukes? Would you rely on it being less dangerous than that?
When I imagine that I could make my copy which would be identical to me, sharing my goals, able to copy its experiences back to me, and willing to die for me (something like Naruto’s clones), taking over the society seems rather easy. (Assuming that no one else has this ability, and no one suspects me of having it. In real life it would probably help if all the clones looked different, but had an ability to recognize each other.)
Research: For each interesting topic I could make dozen clones which would study the topic in libraries and universities, and discuss their findings with each other. I don’t suppose it would make me an expert on everything, but I could get at least all the university-level education on most things.
Resources: If I can make more money than I spend, and if I don’t make too much copies to imbalance the economy, I can let a few dozen clones work and produce the money for the rest of them. At least in the starting phase, until my research groups discover better ways to make money.
Contacts: Different clones could go to different places making useful contacts wil different kinds of people. Sometimes you find a person which can help your goals significantly. With many clones I could make contacts in many different social groups, and overcome language or religious barriers (I can have a few clones learn the language or join the religion).
Multiple “first impressions”: If I need a help of a given person or organization, I could in many cases gain their trust by sending multiple different clones to them, using different strategies to befriend them, until I find one that works.
Taking over democratic organizations: Any organization with low barriers to entry and democratic voting can be taken over by sending enough clones there, and then voting some of the clones as new leaders. A typical non-governmental organization or even a smaller political party could be gained this way. I don’t even need a majority of clones there: two potential leaders competing with each other, half dozen experts openly supporting each of them, and dozen people befriending random voters and explaining them why leader X or leader Y is the perfect choice; then most of the voting would be done by other people.
Assassination: If someone is too much of a problem, I can create a clone which kills them and then disappears. This should be used very rarely, not to draw attention to my abilities.
Safety: To protect myself, I would send my different clones to different countries over the world.
Joining all the winning sides: If there is an important group of people, I could join them, even the groups fighting against each other. Whoever wins, some of my clones are on the winning side.
There are a lot of “ifs”, though.
If that AI runs on expensive or specialized hardware, it can’t necessarily expand much. For instance, if it runs on hardware worth millions of dollars, it can’t exactly copy itself just anywhere yet. Assuming that the first AI of that level will be cutting edge research and won’t be cheap, that gives a certain time window to study it safely.
The AI may be dangerous if it appeared now, but if it appears in, say, fifty years, then it will have to deal with the state of the art fifty years from now. Expanding without getting caught might be considerably more difficult then than it is now—weak AI will be all over the place, for one.
Last, but not least, the AI must have access to its own source code in order to copy it. That’s far from a given, especially if it’s a neural architecture. A human-level AI would not know how it works any more than we know how we work, so if it has no read access to itself or no way to probe its own circuitry, it won’t be able to copy itself at all. I doubt the first AI would actually have fine-grained access to its own inner workings, and I doubt it would have anywhere close to the amount of resources required to reverse engineer itself. Of course, that point is moot if some fool does give it access...
I agree with your first point, though it gets worse for us as hardware gets cheaper and cheaper.
I like your second point even more: it’s actionable. We could work on the security of personal computers.
That last one is incorrect however. The AI only have to access its object code in order to copy itself. That’s something even current computer viruses can do. And we’re back to boxing it.
If the AI is a learning system such as a neural network, and I believe that’s quite likely to be the case, there is no source/object dichotomy at all and the code may very well be unreadable outside of simple local update procedures that are completely out of the AI’s control. In other words, it might be physically impossible for both the AI and ourselves to access the AI’s object code—it would be locked in a hardware box with no physical wires to probe its contents, basically.
I mean, think of a physical hardware circuit implementing a kind of neuron network—in order for the network to be “copiable”, you need to be able to read the values of all neurons. However, that requires a global clock (to ensure synchronization, though AI might tolerate being a bit out of phase) and a large number of extra wires connecting each component to busses going out of the system. Of course, all that extra fluff inflates the cost of the system, makes it bigger, slower and probably less energy efficient. Since the first human-level AI won’t just come out of nowhere, it will probably use off-the-shelf digital neural components, and for cost and speed reasons, these components might not actually offer any way to copy their contents.
This being said, even if the AI runs on conventional hardware, locking it out of its own object code isn’t exactly rocket science. The specification of some programming languages already guarantee that this cannot happen, and type/proof theory is an active research field that may very well be able to prove the conformance of implementation to specification. If the AI is a neural network emulated on conventional hardware, the risks that it can read itself without permission are basically zilch.
What we usually mean by intelligence doesn’t include the skills necessary for getting to be in charge.
There are various notions of intelligence, social intelligence includes the skills for getting in charge.
My point is that human-level intelligence, even replicated or sped up, is generally not enough.