But an AGI, whether FAI or uFAI, will be the last program that humans get to write and execute unsupervised. We will not get to issue patches.
In fiction, yes. Fictional technology appears overnight, works the first time without requiring continuing human effort for debugging and maintenance, and can do all sorts of wondrous things.
In real life, the picture is very different. Real life technology has a small fraction of the capabilities of its fictional counterpart, and is developed incrementally, decade by painfully slow decade. If intelligent machines ever actually come into existence, not only will there be plenty of time to issue patches, but patching will be precisely the process by which they are developed in the first place.
I agree somewhat with this as a set of conclusions, but your argument deserves to get downvoted because you’ve made statements that are highly controversial. The primary issue is that, if one thinks that an AI can engage in recursive self-improvement and can do so quickly, then once there’s an AI that’s at all capable of such improvement, the AI will rapidly move outside our control. There are arguments against such a possibility being likely, but this is not a trivial matter. Moreover, comparing the situation to fiction is unhelpful- just because something is common in fiction that’s not an argument that such a situation can’t actually happen in practice. Reversed stupidity is not intelligence.
I read the subtext as ”...you’ve made statements that are highly controversial without attempting to support them”. Suggesting that there will be plenty of time to debug, maintain, and manually improve anything that actually fits the definition of “AGI” is a very significant disagreement with some fairly standard LW conclusions, and it may certainly be stated, but not as a casual assumption or a fact; it should be accompanied by an accordingly serious attempt to justify it.
To be sure, the fact that something is commonplace in fiction doesn’t prove it false. What it does show is that we should distrust our intuition on it, because it’s clearly an idea to which we are positively disposed regardless of its truth value—in the Bayesian sense, that is evidence against it.
The stronger argument against something is of course its consistent failure to occur in real life. The entire history of technological development says that technology in the real world does not work the way it would need to for the ‘AI go foom’ scenario. If 100% evidence against and 0% evidence for a proposition should not be enough to get us to disbelieve it, then what should?
Not to mention that when you look at the structure of the notion of recursive self-improvement, it doesn’t even make sense. A machine is not going to be able to completely replace human programmers until it is smarter than even the smartest humans in every relevant sense, which given the differences in architecture, is an extraordinarily stringent criterion, and one far beyond anything unaided humans could ever possibly build. If such an event ever comes about in the very distant future, it will necessarily follow a long path of development in which AI is used to create generation after generation of improved tools in an extended bootstrapping process that has yet to even get started.
And indeed this is not a trivial matter—if people start basing decisions on the ‘AI go foom’ belief, that’s exactly the kind of thing that could snuff out whatever chance of survival and success we might have had.
Re: “The primary issue is that, if one thinks that an AI can engage in recursive self-improvement and can do so quickly, then once there’s an AI that’s at all capable of such improvement, the AI will rapidly move outside our control.”
If its creators are incompetent. Those who think this are essentially betting on the incompetence of the creators.
There are numerous counter-arguments—the shifting moral zeitgeist, the downward trend in deliberate death, the safety record of previous risky tech enterprises.
A stop button seems like a relatively simple and effective safely feature. If you can get the machine to do anything at all, then you can probably get it to turn itself off.
The creators will likely be very smart humans assisted by very smart machines. Betting on their incompetence is not a particularly obvious thing to do.
Missing the point. I wasn’t arguing that there aren’t reasons to think that the bad AI goes FOOM won’t happen. Indeed, I said explicitly that I didn’t think it would occur. My point was that if one is going to make an argument that relies on that here one needs to be aware that the premise is controversial and be clear about that (say giving basic reasoning for it, or even just saying “If one accepts that X then...” etc.).
In fiction, yes. Fictional technology appears overnight, works the first time without requiring continuing human effort for debugging and maintenance, and can do all sorts of wondrous things.
In real life, the picture is very different. Real life technology has a small fraction of the capabilities of its fictional counterpart, and is developed incrementally, decade by painfully slow decade. If intelligent machines ever actually come into existence, not only will there be plenty of time to issue patches, but patching will be precisely the process by which they are developed in the first place.
I agree somewhat with this as a set of conclusions, but your argument deserves to get downvoted because you’ve made statements that are highly controversial. The primary issue is that, if one thinks that an AI can engage in recursive self-improvement and can do so quickly, then once there’s an AI that’s at all capable of such improvement, the AI will rapidly move outside our control. There are arguments against such a possibility being likely, but this is not a trivial matter. Moreover, comparing the situation to fiction is unhelpful- just because something is common in fiction that’s not an argument that such a situation can’t actually happen in practice. Reversed stupidity is not intelligence.
Did you accidentally pick the wrong adjective, or did you seriously mean that controversy is unwelcome in LW comment threads?
I read the subtext as ”...you’ve made statements that are highly controversial without attempting to support them”. Suggesting that there will be plenty of time to debug, maintain, and manually improve anything that actually fits the definition of “AGI” is a very significant disagreement with some fairly standard LW conclusions, and it may certainly be stated, but not as a casual assumption or a fact; it should be accompanied by an accordingly serious attempt to justify it.
No. See ata’s reply which summarizes exactly what I meant.
To be sure, the fact that something is commonplace in fiction doesn’t prove it false. What it does show is that we should distrust our intuition on it, because it’s clearly an idea to which we are positively disposed regardless of its truth value—in the Bayesian sense, that is evidence against it.
The stronger argument against something is of course its consistent failure to occur in real life. The entire history of technological development says that technology in the real world does not work the way it would need to for the ‘AI go foom’ scenario. If 100% evidence against and 0% evidence for a proposition should not be enough to get us to disbelieve it, then what should?
Not to mention that when you look at the structure of the notion of recursive self-improvement, it doesn’t even make sense. A machine is not going to be able to completely replace human programmers until it is smarter than even the smartest humans in every relevant sense, which given the differences in architecture, is an extraordinarily stringent criterion, and one far beyond anything unaided humans could ever possibly build. If such an event ever comes about in the very distant future, it will necessarily follow a long path of development in which AI is used to create generation after generation of improved tools in an extended bootstrapping process that has yet to even get started.
And indeed this is not a trivial matter—if people start basing decisions on the ‘AI go foom’ belief, that’s exactly the kind of thing that could snuff out whatever chance of survival and success we might have had.
Re: “The primary issue is that, if one thinks that an AI can engage in recursive self-improvement and can do so quickly, then once there’s an AI that’s at all capable of such improvement, the AI will rapidly move outside our control.”
If its creators are incompetent. Those who think this are essentially betting on the incompetence of the creators.
There are numerous counter-arguments—the shifting moral zeitgeist, the downward trend in deliberate death, the safety record of previous risky tech enterprises.
A stop button seems like a relatively simple and effective safely feature. If you can get the machine to do anything at all, then you can probably get it to turn itself off.
See: http://alife.co.uk/essays/stopping_superintelligence/
The creators will likely be very smart humans assisted by very smart machines. Betting on their incompetence is not a particularly obvious thing to do.
Missing the point. I wasn’t arguing that there aren’t reasons to think that the bad AI goes FOOM won’t happen. Indeed, I said explicitly that I didn’t think it would occur. My point was that if one is going to make an argument that relies on that here one needs to be aware that the premise is controversial and be clear about that (say giving basic reasoning for it, or even just saying “If one accepts that X then...” etc.).