I think we need to be very careful before extrapolating from primitive elevator control systems to superintelligent AI.
I am extrapolating from a general trend, and not specific systems. The general trend is that newer generations of software less frequently crash or exhibit unexpected side-effects (just look at Windows 95 vs. Windows 8).
If we want to ever be able to build an AI that can take over the world then we will need to become really good at either predicting how software works or at spotting errors. In other words, if IBM Watson would have started singing, or if it got stuck on a query, then it would have lost at Jeopardy. But this trend contradicts the idea of an AI killing all humans in order to calculate 1+1. If we are bad enough at software engineering to miss such failure modes then we won’t be good enough to enable our software to take over the world.
In other words, you’re saying that if someone is smart enough to build a superintelligent AI, she should be smart enough it make it friendly.
Well, firstly this claim doesn’t imply we should be researching FAI and/or that MIRI’s work is superfluous. It just implies that nobody will build a superintelligent AI before the problem of friendliness is solved.
Secondly, I’m not at all convinced this claim is true. It sounds like saying “if they are smart enough to build the Chernobyl nuclear power plant, they are smart enough to make it safe”. But they weren’t.
Improvement in software quality is probably due to improvement in design and testing methodologies and tools, response to increasing market expectations etc. I wouldn’t count on these effects to safe-guard against an existential catastrophe. If a piece of software is buggy, it becomes less likely to be released. If an AI has a poorly designed utility function but a perfectly designed decision engine, there might be no time to pull the plug. The product manager won’t stop the release because the software will release itself.
If growth of intelligence due to self-improvement is a slow process than the creators of the AI will have time to respond and fix the problems. However, if “AI foom” is real, they won’t have time to do it. One moment it’s a harmless robot driving around the room and building castles from colorful cubes. Another moment the whole galaxy is on its way to become a pile of toy castles.
The engineers who build the first superintelligent AI might simply lack the imagination to believe it will really become superintelligent. Imagine one of them inventing a genius mathematical theory of self-improving intelligent systems. Suppose she never heard about AI existential risks etc. Will she automatically think “hmm, once I implement this theory the AI will become so powerful it will paperclip the universe”? I seriously doubt it. More likely it would be “wow, that formula came out really neat, I wonder how good my software will become once I code it in”. I know I would think it. But then, maybe I’m just too stupid to build an AGI...
I am extrapolating from a general trend, and not specific systems. The general trend is that newer generations of software less frequently crash or exhibit unexpected side-effects (just look at Windows 95 vs. Windows 8).
If we want to ever be able to build an AI that can take over the world then we will need to become really good at either predicting how software works or at spotting errors. In other words, if IBM Watson would have started singing, or if it got stuck on a query, then it would have lost at Jeopardy. But this trend contradicts the idea of an AI killing all humans in order to calculate 1+1. If we are bad enough at software engineering to miss such failure modes then we won’t be good enough to enable our software to take over the world.
In other words, you’re saying that if someone is smart enough to build a superintelligent AI, she should be smart enough it make it friendly.
Well, firstly this claim doesn’t imply we should be researching FAI and/or that MIRI’s work is superfluous. It just implies that nobody will build a superintelligent AI before the problem of friendliness is solved.
Secondly, I’m not at all convinced this claim is true. It sounds like saying “if they are smart enough to build the Chernobyl nuclear power plant, they are smart enough to make it safe”. But they weren’t.
Improvement in software quality is probably due to improvement in design and testing methodologies and tools, response to increasing market expectations etc. I wouldn’t count on these effects to safe-guard against an existential catastrophe. If a piece of software is buggy, it becomes less likely to be released. If an AI has a poorly designed utility function but a perfectly designed decision engine, there might be no time to pull the plug. The product manager won’t stop the release because the software will release itself.
If growth of intelligence due to self-improvement is a slow process than the creators of the AI will have time to respond and fix the problems. However, if “AI foom” is real, they won’t have time to do it. One moment it’s a harmless robot driving around the room and building castles from colorful cubes. Another moment the whole galaxy is on its way to become a pile of toy castles.
The engineers who build the first superintelligent AI might simply lack the imagination to believe it will really become superintelligent. Imagine one of them inventing a genius mathematical theory of self-improving intelligent systems. Suppose she never heard about AI existential risks etc. Will she automatically think “hmm, once I implement this theory the AI will become so powerful it will paperclip the universe”? I seriously doubt it. More likely it would be “wow, that formula came out really neat, I wonder how good my software will become once I code it in”. I know I would think it. But then, maybe I’m just too stupid to build an AGI...