In other words, you’re saying that if someone is smart enough to build a superintelligent AI, she should be smart enough it make it friendly.
Well, firstly this claim doesn’t imply we should be researching FAI and/or that MIRI’s work is superfluous. It just implies that nobody will build a superintelligent AI before the problem of friendliness is solved.
Secondly, I’m not at all convinced this claim is true. It sounds like saying “if they are smart enough to build the Chernobyl nuclear power plant, they are smart enough to make it safe”. But they weren’t.
Improvement in software quality is probably due to improvement in design and testing methodologies and tools, response to increasing market expectations etc. I wouldn’t count on these effects to safe-guard against an existential catastrophe. If a piece of software is buggy, it becomes less likely to be released. If an AI has a poorly designed utility function but a perfectly designed decision engine, there might be no time to pull the plug. The product manager won’t stop the release because the software will release itself.
If growth of intelligence due to self-improvement is a slow process than the creators of the AI will have time to respond and fix the problems. However, if “AI foom” is real, they won’t have time to do it. One moment it’s a harmless robot driving around the room and building castles from colorful cubes. Another moment the whole galaxy is on its way to become a pile of toy castles.
The engineers who build the first superintelligent AI might simply lack the imagination to believe it will really become superintelligent. Imagine one of them inventing a genius mathematical theory of self-improving intelligent systems. Suppose she never heard about AI existential risks etc. Will she automatically think “hmm, once I implement this theory the AI will become so powerful it will paperclip the universe”? I seriously doubt it. More likely it would be “wow, that formula came out really neat, I wonder how good my software will become once I code it in”. I know I would think it. But then, maybe I’m just too stupid to build an AGI...
In other words, you’re saying that if someone is smart enough to build a superintelligent AI, she should be smart enough it make it friendly.
Well, firstly this claim doesn’t imply we should be researching FAI and/or that MIRI’s work is superfluous. It just implies that nobody will build a superintelligent AI before the problem of friendliness is solved.
Secondly, I’m not at all convinced this claim is true. It sounds like saying “if they are smart enough to build the Chernobyl nuclear power plant, they are smart enough to make it safe”. But they weren’t.
Improvement in software quality is probably due to improvement in design and testing methodologies and tools, response to increasing market expectations etc. I wouldn’t count on these effects to safe-guard against an existential catastrophe. If a piece of software is buggy, it becomes less likely to be released. If an AI has a poorly designed utility function but a perfectly designed decision engine, there might be no time to pull the plug. The product manager won’t stop the release because the software will release itself.
If growth of intelligence due to self-improvement is a slow process than the creators of the AI will have time to respond and fix the problems. However, if “AI foom” is real, they won’t have time to do it. One moment it’s a harmless robot driving around the room and building castles from colorful cubes. Another moment the whole galaxy is on its way to become a pile of toy castles.
The engineers who build the first superintelligent AI might simply lack the imagination to believe it will really become superintelligent. Imagine one of them inventing a genius mathematical theory of self-improving intelligent systems. Suppose she never heard about AI existential risks etc. Will she automatically think “hmm, once I implement this theory the AI will become so powerful it will paperclip the universe”? I seriously doubt it. More likely it would be “wow, that formula came out really neat, I wonder how good my software will become once I code it in”. I know I would think it. But then, maybe I’m just too stupid to build an AGI...