To be sure, the fact that something is commonplace in fiction doesn’t prove it false. What it does show is that we should distrust our intuition on it, because it’s clearly an idea to which we are positively disposed regardless of its truth value—in the Bayesian sense, that is evidence against it.
The stronger argument against something is of course its consistent failure to occur in real life. The entire history of technological development says that technology in the real world does not work the way it would need to for the ‘AI go foom’ scenario. If 100% evidence against and 0% evidence for a proposition should not be enough to get us to disbelieve it, then what should?
Not to mention that when you look at the structure of the notion of recursive self-improvement, it doesn’t even make sense. A machine is not going to be able to completely replace human programmers until it is smarter than even the smartest humans in every relevant sense, which given the differences in architecture, is an extraordinarily stringent criterion, and one far beyond anything unaided humans could ever possibly build. If such an event ever comes about in the very distant future, it will necessarily follow a long path of development in which AI is used to create generation after generation of improved tools in an extended bootstrapping process that has yet to even get started.
And indeed this is not a trivial matter—if people start basing decisions on the ‘AI go foom’ belief, that’s exactly the kind of thing that could snuff out whatever chance of survival and success we might have had.
To be sure, the fact that something is commonplace in fiction doesn’t prove it false. What it does show is that we should distrust our intuition on it, because it’s clearly an idea to which we are positively disposed regardless of its truth value—in the Bayesian sense, that is evidence against it.
The stronger argument against something is of course its consistent failure to occur in real life. The entire history of technological development says that technology in the real world does not work the way it would need to for the ‘AI go foom’ scenario. If 100% evidence against and 0% evidence for a proposition should not be enough to get us to disbelieve it, then what should?
Not to mention that when you look at the structure of the notion of recursive self-improvement, it doesn’t even make sense. A machine is not going to be able to completely replace human programmers until it is smarter than even the smartest humans in every relevant sense, which given the differences in architecture, is an extraordinarily stringent criterion, and one far beyond anything unaided humans could ever possibly build. If such an event ever comes about in the very distant future, it will necessarily follow a long path of development in which AI is used to create generation after generation of improved tools in an extended bootstrapping process that has yet to even get started.
And indeed this is not a trivial matter—if people start basing decisions on the ‘AI go foom’ belief, that’s exactly the kind of thing that could snuff out whatever chance of survival and success we might have had.