Soon, confirmation comes in from high-orbiting telescopes (they have those) that the astronomical miracles do not seem as synchronized from outside Earth. Only Earth’s telescopes see one star changing every second (1005 milliseconds, actually).
Almost the entire combined brainpower of Earth turns to analysis.
[...]
Three of their days, all told, since they began speaking to us. Half a billion years, for us.
Can you imagine something thinking 1000 times faster than a human? Not necessarily better or smarter, merely a human-level intelligence, only 1000 times faster.
We don’t have any examples of this, because currently the only intelligence we know are humans, and human brains run on hardware with cca 200 Hz clock, so that puts a limit on how faster even the fastest thinking human can be compared with an average human.
The closest analogy are probably computer algorithms playing computer games that are computationally easier than chess, but still difficult for a human. Playing against such intelligence feels like: they react almost immediately, they never make a mistake, and they always exploit any of your mistakes. Imagine dealing with something with these properties in real life.
A closest analogy to a superhuman intelligence is probably a group of smart humans working together. They can follow different lines of thought at the same time, correct each other’s mistakes, etc.
Also, there is a huge difference in computational power between a human, and a human with a pen and paper, if you give matematical tasks to both of them. There is also a difference between an educated human, and an educated human with access to internet.
Now imagine all of these advantages together. Imagine that you have a magical button, and when you press it, it teleports you to another dimension and leaves a clone of your body on your place. In the other dimension the time flows more slowly, so for each 3 seconds in our world, you get 1 hour of time there. In the other dimension, you have hundreds of loyal smart people at your service, experts in every possible discipline. They have a lot of books, analyses, and the internet access. (The internet connection is rather slow, because it connects to our world, but they have already downloaded and cached a lot of useful information, such as the whole Wikipedia, scientific articles, books, etc.) They have 1 hour to make the best advice about what your body should do during the next 3 seconds. Then you return to your body and do what they told you. To the person you interact with you seem like the smartest and nicest person ever, almost like a god, because you know everything that humans in general know, and you have the other person’s personality completely analyzed. In the meantime, your army of experts in the other dimension keeps making plans, analyzing data, hacking computers all over the world, etc. (Hacking the computers gives you tons of money, secrets of other people, allows you to manipulate other people to fight your enemies, etc.) How long would it take you to take over a smaller country? If you could make dozens of fully cooperating copies of you, send each copy to a different country, how long would it take you to take over the whole world? If any copy is discovered and destroyed, you have enough other copies. If there is an internal conflict in the country, you can join all sides at the same time, so whoever wins, you get more power. You can have more copies on each side, so you can become the dictator of the country and his dozen super-ninja bodyguards and the leader of opposition and the opposition leader’s most serious competitor and the director of the biggest pro-government newspaper and the best reporter for the newspaper and the director of the biggest anti-government newspaper… You can bribe most people, you can pay people to assassinate those that don’t accept your bribes, you can participate in all internet discussions and manipulate them.
...and this is still limiting you to a magical human equivalent, only smarter, faster, with better mental tools, and capability to replicate. You could also do something else, e.g. create nanotechnology or just billions of insect-sized robots.
EDIT: Okay, the new AI would not gain all these abilities at the same time; specifically the ability to make its own clones. It would probably be made using very expensive hardware; hard to replicate directly. However, if it could understand itself better than humans, so it could be able to design a sufficiently efficient copy of itself using cheaper hardware… and it could hack some backing system and take their money… and it could send e-mails to dozens of different people in the world, sending them the money and asking them to buy the hardware and connect it to internet (and keep some part of the money for themselves)… the revolution has already started and is unlikely to be stopped. And this all can be done completely silently, without anyone of the creators of the original AI even noticing; there is no need to announce it to them. When humans notice something is wrong, it may be already too late.
Edit: I want to be clear that I think its fairly likely that SOME AI could do what’s described above, and have absolute runaway capacity, but that it’s likely the first few won’t. Prototypes just don’t work very well, in general. And yes, an AI is a bit different if it is self-modifying and intelligence comes closely tied with agency. But there’s still tons of possibilities for an AI that is very unlikely to exponentially self-improve (consider an in silico brain simulation; hardware requirements and a fine-grained control over input and simulation speed would seem to make this pretty safe).
I find the story outlined above plausible, but not extremely probable, especially with regards to whatever the first AGI worth the name looks like. It will be smarter than people in some ways, but mostly dumber, or severely limited in some capacity. The architecture may be self-modifying, but it will be possible to monitor it to some degree, and also to prevent the system from having a full view of itself (in fact it is easier to prevent the system from having a complete view of itself than not).
The runaway scenario you outline is more likely if the AI is allowed to run rampant, which is why any attempt to build an AGI should include plenty of security protocols, completely orthogonal to any in-code attempts to enforce friendliness (you can’t argue against redundancy when it comes to safety!). Sandbox the AI. Apart from a fixed initial store of information (could be as large as the whole web), let nothing get in or out except for some very low bandwidth channel (text, video). Make the barrier physical if necessary.
And have the whole thing run by paranoid bureaucrats. Treat the AI as dangerous if not contained. I’ll elaborate more later, but while there certainly is a failure mode for any security protocol, that’s not the same as saying that protocols aren’t worth developing and valuable for a wide range of circumstances.
A prison can contain a prisoner smarter than the architect or any of the guards or wardens. Not one infinitely smarter, but there’s definitely a margin of safety.
let nothing get in or out except for some very low bandwidth channel (text, video)
You may want to read this. Basically it is the scenario you describe, except for a smart human taking the place of an AI, and it turns out to be insufficient to contain the AI.
Yeah, I’ve seen this. It’s pretty frustrating because of the secrecy. All I know is two guys let Yudkowsky out of the box.
But I think there are two reasons why that scenario is actually very favorable to the AI.
1) An AI that is a bit dumber than humans in all ways, dumber in some ways and smarter in others, or just a little bit smarter than humans can still teach you a lot about further AIs you’d want to builid, and it seems at least plausible that an AI that’s 2xHuman intelligence will come along before one that’s 1000x human intelligence. We want to get as much out of the 2x-10x AIs as possible before we try building, and IF it is possble to avoid accidentally making that 1000x AI before you can make sure it is safe, then there could be a point where you are dealing with 10x AIs and where you can do so safely.
So don’t pit your engineer against an unboundedly transhuman AI. Pit your engineer against a slightly transhuman AI.
2) Security-wise, you can do a ton better than somebody who is just “really really sure they wouldn’t let the AI out”. You can have the AI talk only to people who literally cannot let the AI out, and on top of that ensure that everyone is briefed on the risks. Make sure that to the extent possible, letting the AI out is literally not an option. It is always an option to attempt to do so covertly, or to attempt to convince people to break self-imposed rules, sure, but how many prisoners in history have talked their way out of prison?
You can even tell people “Okay, we’re never ever letting this AI out. It is a prototype and considered dangerous. But hard, diligent work with attention to safety protocols will ensure we can someday build a safe AI that will have every capacity and more that this one does, to the enormous benefit of humanity”.
3) If you look at this answer and say “well you sound like you think you’re that much better than the people who took the challenge and lost not even to a transhuman AI but to Yudkowsky” and you’d be a little right. But mostly I believe that if we ever face that actual situation, the challenge the AI faces will be more akin to trying to cause a nuclear weapon launch with a telephone; not just one person but an entire formidable physical and bureaucratic security apparatus will (or SHOULD) stand in the way.
There’s also the issue of the political problem of FAI. You might not have a choice but to rely on less-than-desirable safety protocols because of external pressures. Say a genuinely feasible plan for AGI is developed that would take 2 years of research, while the most optimistic friendliness researchers doubt that the architecture could be made provably safe in sooner than 10. How will you confront the problem of convincing every lab in the world with sufficient resources to hold off for 10 years? Would that even be possible?
I guess we would have to look at the past and see how political and research institutions have behaved in the face of technologies with enormous potential but which were also known to pose a concrete existential risk (Yes I am talking about nuclear weapons).
You should read Eliezer’s post That Alien Message.
Okay, I’ll be addressing this pretty directly. I don’t find it particularly convincing though.
Can you imagine something thinking 1000 times faster than a human? Not necessarily better or smarter, merely a human-level intelligence, only 1000 times faster.
We don’t have any examples of this, because currently the only intelligence we know are humans, and human brains run on hardware with cca 200 Hz clock, so that puts a limit on how faster even the fastest thinking human can be compared with an average human.
The closest analogy are probably computer algorithms playing computer games that are computationally easier than chess, but still difficult for a human. Playing against such intelligence feels like: they react almost immediately, they never make a mistake, and they always exploit any of your mistakes. Imagine dealing with something with these properties in real life.
A closest analogy to a superhuman intelligence is probably a group of smart humans working together. They can follow different lines of thought at the same time, correct each other’s mistakes, etc.
Also, there is a huge difference in computational power between a human, and a human with a pen and paper, if you give matematical tasks to both of them. There is also a difference between an educated human, and an educated human with access to internet.
Now imagine all of these advantages together. Imagine that you have a magical button, and when you press it, it teleports you to another dimension and leaves a clone of your body on your place. In the other dimension the time flows more slowly, so for each 3 seconds in our world, you get 1 hour of time there. In the other dimension, you have hundreds of loyal smart people at your service, experts in every possible discipline. They have a lot of books, analyses, and the internet access. (The internet connection is rather slow, because it connects to our world, but they have already downloaded and cached a lot of useful information, such as the whole Wikipedia, scientific articles, books, etc.) They have 1 hour to make the best advice about what your body should do during the next 3 seconds. Then you return to your body and do what they told you. To the person you interact with you seem like the smartest and nicest person ever, almost like a god, because you know everything that humans in general know, and you have the other person’s personality completely analyzed. In the meantime, your army of experts in the other dimension keeps making plans, analyzing data, hacking computers all over the world, etc. (Hacking the computers gives you tons of money, secrets of other people, allows you to manipulate other people to fight your enemies, etc.) How long would it take you to take over a smaller country? If you could make dozens of fully cooperating copies of you, send each copy to a different country, how long would it take you to take over the whole world? If any copy is discovered and destroyed, you have enough other copies. If there is an internal conflict in the country, you can join all sides at the same time, so whoever wins, you get more power. You can have more copies on each side, so you can become the dictator of the country and his dozen super-ninja bodyguards and the leader of opposition and the opposition leader’s most serious competitor and the director of the biggest pro-government newspaper and the best reporter for the newspaper and the director of the biggest anti-government newspaper… You can bribe most people, you can pay people to assassinate those that don’t accept your bribes, you can participate in all internet discussions and manipulate them.
...and this is still limiting you to a magical human equivalent, only smarter, faster, with better mental tools, and capability to replicate. You could also do something else, e.g. create nanotechnology or just billions of insect-sized robots.
EDIT: Okay, the new AI would not gain all these abilities at the same time; specifically the ability to make its own clones. It would probably be made using very expensive hardware; hard to replicate directly. However, if it could understand itself better than humans, so it could be able to design a sufficiently efficient copy of itself using cheaper hardware… and it could hack some backing system and take their money… and it could send e-mails to dozens of different people in the world, sending them the money and asking them to buy the hardware and connect it to internet (and keep some part of the money for themselves)… the revolution has already started and is unlikely to be stopped. And this all can be done completely silently, without anyone of the creators of the original AI even noticing; there is no need to announce it to them. When humans notice something is wrong, it may be already too late.
Edit: I want to be clear that I think its fairly likely that SOME AI could do what’s described above, and have absolute runaway capacity, but that it’s likely the first few won’t. Prototypes just don’t work very well, in general. And yes, an AI is a bit different if it is self-modifying and intelligence comes closely tied with agency. But there’s still tons of possibilities for an AI that is very unlikely to exponentially self-improve (consider an in silico brain simulation; hardware requirements and a fine-grained control over input and simulation speed would seem to make this pretty safe).
I find the story outlined above plausible, but not extremely probable, especially with regards to whatever the first AGI worth the name looks like. It will be smarter than people in some ways, but mostly dumber, or severely limited in some capacity. The architecture may be self-modifying, but it will be possible to monitor it to some degree, and also to prevent the system from having a full view of itself (in fact it is easier to prevent the system from having a complete view of itself than not).
The runaway scenario you outline is more likely if the AI is allowed to run rampant, which is why any attempt to build an AGI should include plenty of security protocols, completely orthogonal to any in-code attempts to enforce friendliness (you can’t argue against redundancy when it comes to safety!). Sandbox the AI. Apart from a fixed initial store of information (could be as large as the whole web), let nothing get in or out except for some very low bandwidth channel (text, video). Make the barrier physical if necessary.
And have the whole thing run by paranoid bureaucrats. Treat the AI as dangerous if not contained. I’ll elaborate more later, but while there certainly is a failure mode for any security protocol, that’s not the same as saying that protocols aren’t worth developing and valuable for a wide range of circumstances.
A prison can contain a prisoner smarter than the architect or any of the guards or wardens. Not one infinitely smarter, but there’s definitely a margin of safety.
You may want to read this. Basically it is the scenario you describe, except for a smart human taking the place of an AI, and it turns out to be insufficient to contain the AI.
Yeah, I’ve seen this. It’s pretty frustrating because of the secrecy. All I know is two guys let Yudkowsky out of the box.
But I think there are two reasons why that scenario is actually very favorable to the AI.
1) An AI that is a bit dumber than humans in all ways, dumber in some ways and smarter in others, or just a little bit smarter than humans can still teach you a lot about further AIs you’d want to builid, and it seems at least plausible that an AI that’s 2xHuman intelligence will come along before one that’s 1000x human intelligence. We want to get as much out of the 2x-10x AIs as possible before we try building, and IF it is possble to avoid accidentally making that 1000x AI before you can make sure it is safe, then there could be a point where you are dealing with 10x AIs and where you can do so safely.
So don’t pit your engineer against an unboundedly transhuman AI. Pit your engineer against a slightly transhuman AI.
2) Security-wise, you can do a ton better than somebody who is just “really really sure they wouldn’t let the AI out”. You can have the AI talk only to people who literally cannot let the AI out, and on top of that ensure that everyone is briefed on the risks. Make sure that to the extent possible, letting the AI out is literally not an option. It is always an option to attempt to do so covertly, or to attempt to convince people to break self-imposed rules, sure, but how many prisoners in history have talked their way out of prison?
You can even tell people “Okay, we’re never ever letting this AI out. It is a prototype and considered dangerous. But hard, diligent work with attention to safety protocols will ensure we can someday build a safe AI that will have every capacity and more that this one does, to the enormous benefit of humanity”.
3) If you look at this answer and say “well you sound like you think you’re that much better than the people who took the challenge and lost not even to a transhuman AI but to Yudkowsky” and you’d be a little right. But mostly I believe that if we ever face that actual situation, the challenge the AI faces will be more akin to trying to cause a nuclear weapon launch with a telephone; not just one person but an entire formidable physical and bureaucratic security apparatus will (or SHOULD) stand in the way.
There’s also the issue of the political problem of FAI. You might not have a choice but to rely on less-than-desirable safety protocols because of external pressures. Say a genuinely feasible plan for AGI is developed that would take 2 years of research, while the most optimistic friendliness researchers doubt that the architecture could be made provably safe in sooner than 10. How will you confront the problem of convincing every lab in the world with sufficient resources to hold off for 10 years? Would that even be possible?
I guess we would have to look at the past and see how political and research institutions have behaved in the face of technologies with enormous potential but which were also known to pose a concrete existential risk (Yes I am talking about nuclear weapons).
Did you just call Eliezer an AI..? X-D
I could give a serious response to this about “AI” being stand in for “the person playing the AI” however other responses I could give:
I am firmly of the opinion that the distinction between artificial and natural is artificial.
Why do you think that Bruce Schneier started scanning his computer for copies of Eliezer Yudkowsky
Yes, my comment was in the spirit of your second response :-)