There is a large, continuous spectrum between making an AI and hoping it works out okay, and waiting for a formal proof of friendliness. Now, I don’t think a complete proof is feasible; we’ve never managed a formal proof for anything close to that level of complexity, and the proof would be as likely to contain bugs as the program would. However, that doesn’t mean we shouldn’t push in that direction. Current practice in AI research seems to be to publish everything and take no safety precautions whatsoever, and that is definitely not good.
Suppose an AGI is created, initially not very smart but capable of rapid improvement, either with further development by humans or by giving it computing resources and letting it self-improve.Suppose, further, that its creators publish the source code, or allow it to be leaked or stolen.
AI improvement will probably proceed in a series of steps: the AI designs a successor, spends some time inspecting it to make sure the successor has the same values, then hands over control, then repeat. At each stage, the same tradeoff between speed and safety applies: more time spent verifying the successor means a lower probability of error, but a higher probability that other bad things will happen in the mean time.
And here’s where there’s a real problem. If there’s only one AI improving itself, then it can proceed slowly, knowing that the probability of an asteroid strike, nuclear war or other existential risk is reasonably low. But if there are several AIs doing this at once, then whichever one proceeds least cautiously wins. That situation creates a higher risk of paperclippers, as compared to if there were only one AI developed in secret.
Current practice in AI research seems to be to publish everything and take no safety precautions whatsoever, and that is definitely not good.
Most of the compaines involved (e.g. Google, James Harris Simons) publish little or nothing relating so their code in this area publicly—and few know what safeguards they employ. The government security agencies potentially involved (e.g. the NSA) are even more secretive.
Simons is an AI researcher? News to me. Clearly his fund uses machine learning, but there is an ocean between that and AGI (besides plenty of funds use ML also, DE Shaw and many others).
There is a large, continuous spectrum between making an AI and hoping it works out okay, and waiting for a formal proof of friendliness.
Exactly this!
I think there is a U-shaped response curve to risk versus rigor. Too little rigor ensures disaster, but too much rigor ensures a low rigor alternative is completed first.
When discussing the correct course of action, I think it is critical to consider not just probability of success but also time to success. So far as I’ve seen arguments in favor of SIAI’s course of action have completely ignored this essential aspect of the decision problem.
There is a large, continuous spectrum between making an AI and hoping it works out okay, and waiting for a formal proof of friendliness. Now, I don’t think a complete proof is feasible; we’ve never managed a formal proof for anything close to that level of complexity, and the proof would be as likely to contain bugs as the program would. However, that doesn’t mean we shouldn’t push in that direction. Current practice in AI research seems to be to publish everything and take no safety precautions whatsoever, and that is definitely not good.
Suppose an AGI is created, initially not very smart but capable of rapid improvement, either with further development by humans or by giving it computing resources and letting it self-improve.Suppose, further, that its creators publish the source code, or allow it to be leaked or stolen.
AI improvement will probably proceed in a series of steps: the AI designs a successor, spends some time inspecting it to make sure the successor has the same values, then hands over control, then repeat. At each stage, the same tradeoff between speed and safety applies: more time spent verifying the successor means a lower probability of error, but a higher probability that other bad things will happen in the mean time.
And here’s where there’s a real problem. If there’s only one AI improving itself, then it can proceed slowly, knowing that the probability of an asteroid strike, nuclear war or other existential risk is reasonably low. But if there are several AIs doing this at once, then whichever one proceeds least cautiously wins. That situation creates a higher risk of paperclippers, as compared to if there were only one AI developed in secret.
Most of the compaines involved (e.g. Google, James Harris Simons) publish little or nothing relating so their code in this area publicly—and few know what safeguards they employ. The government security agencies potentially involved (e.g. the NSA) are even more secretive.
Simons is an AI researcher? News to me. Clearly his fund uses machine learning, but there is an ocean between that and AGI (besides plenty of funds use ML also, DE Shaw and many others).
Exactly this!
I think there is a U-shaped response curve to risk versus rigor. Too little rigor ensures disaster, but too much rigor ensures a low rigor alternative is completed first.
When discussing the correct course of action, I think it is critical to consider not just probability of success but also time to success. So far as I’ve seen arguments in favor of SIAI’s course of action have completely ignored this essential aspect of the decision problem.