There is a nuclear analog for accident risk. A quote from Richard Hamming:
Shortly before the first field test (you realize that no small scale experiment can be done—either you have a critical mass or you do not), a man asked me to check some arithmetic he had done, and I agreed, thinking to fob it off on some subordinate. When I asked what it was, he said, “It is the probability that the test bomb will ignite the whole atmosphere.” I decided I would check it myself! The next day when he came for the answers I remarked to him, “The arithmetic was apparently correct but I do not know about the formulas for the capture cross sections for oxygen and nitrogen—after all, there could be no experiments at the needed energy levels.” He replied, like a physicist talking to a mathematician, that he wanted me to check the arithmetic not the physics, and left. I said to myself, “What have you done, Hamming, you are involved in risking all of life that is known in the Universe, and you do not know much of an essential part?” I was pacing up and down the corridor when a friend asked me what was bothering me. I told him. His reply was, “Never mind, Hamming, no one will ever blame you.”
I don’t really know what this is meant to imply? Maybe you’re answering my question of “did that happen with nukes?”, but I don’t think an affirmative answer means that the analogy starts to work.
I think the nukes-AI analogy is used to argue “people raced to develop nukes despite their downsides, so we should expect the same with AI”; the magnitude/severity of the accident risk is not that relevant to this argument.
I think the nukes-AI analogy is used to argue “people raced to develop nukes despite their downsides, so we should expect the same with AI”
If you’re arguing against that, I’m still not sure what your counter-argument is. To me, the argument is: the upsides of nukes are the ability to take over the world (militarily) and to defend against such attempts. The downsides include risks of local and global catastrophe. People raced to develop nukes because they judged the upsides to be greater than the downsides, in part because they’re not altruists and longtermists. It seems like people will develop potentially unsafe AI for analogous reasons: the upsides include the ability to take over the world (militarily or economically) and to defend against such attempts, and the downsides include risks of local and global catastrophe, and people will likely race to develop AI because they judge the upsides to be greater than the downsides, in part because they’re not altruists and longtermists.
I’m more sympathetic to this argument (which is a claim about what might happen in the future, as opposed to what is happening now, which is the analogy I usually encounter, though possibly not on LessWrong). I still think the analogy breaks down, though in different ways:
There is a strong norm of openness in AI research (though that might be changing). (Though perhaps this was the case with nuclear physics too.)
There is a strong anti-government / anti-military ethic in the AI research community. I’m not sure what the nuclear analog is, but I’m guessing it was neutral or pro-government/military.
Governments are staying a mile away from AGI; their interest in AI is in narrow AI’s applications. Narrow AI applications are diverse, and many can be done by a huge number of people. In contrast, nukes are a single technology, governments were interested in them, and only a few people could plausibly build them. (This is relevant if you think a ton of narrow AI could be used to take over the world economically.)
OpenAI / DeepMind are not adversarial towards each other. In contrast, US / Germany were definitely adversarial.
Assuming you agree that people are already pushing too hard for progress in AGI capability (relative to what’s ideal from a longtermist perspective), I think the current motivations for that are mostly things like money, prestige, scientific curiosity, wanting to make the world a better place (in a misguided/shorttermist way), etc., and not so much wanting to take over the world or to defend against such attempts. This seems likely to persist in the near future, but my concern is that if AGI research gets sufficiently close to fruition, governments will inevitably get involved and start pushing it even harder due to national security considerations. (Recall that Manhattan Project started 8 years before detonation of the first nuke.) Your argument seems more about what’s happening now, and does not really address this concern.
you agree that people are already pushing too hard for progress in AGI capability (relative to what’s ideal from a longtermist perspective)
I’m uncertain, given the potential for AGI to be used to reduce other x-risks. (I don’t have strong opinions on how large other x-risks are and how much potential there is for AGI to differentially help.) But I’m happy to accept this as a premise.
Your argument seems more about what’s happening now, and does not really address this concern.
I think what’s happening now is a good guide into what will happen in the future, at least on short timelines. If AGI is >100 years away, then sure, a lot will change and current facts are relatively unimportant. If it’s < 20 years away, then current facts seem very relevant. I usually focus on the shorter timelines.
For min(20 years, time till AGI), for each individual trend I identified, I’d weakly predict that trend will continue (except perhaps openness, because that’s already changing).
It wasn’t meant as a reply to a particular thing—mainly I’m flagging this as an AI-risk analogy I like.
On that theme, one thing “we don’t know if the nukes will ignite the atmosphere” has in common with AI-risk is that the risk is from reaching new configurations (e.g. temperatures of the sort you get out of a nuclear bomb inside the Earth’s atmosphere) that we don’t have experience with. Which is an entirely different question than “what happens with the nukes after we don’t ignite the atmosphere in a test explosion”.
I like thinking about coordination from this viewpoint.
There is a nuclear analog for accident risk. A quote from Richard Hamming:
https://en.wikipedia.org/wiki/Richard_Hamming#Manhattan_Project
I don’t really know what this is meant to imply? Maybe you’re answering my question of “did that happen with nukes?”, but I don’t think an affirmative answer means that the analogy starts to work.
I think the nukes-AI analogy is used to argue “people raced to develop nukes despite their downsides, so we should expect the same with AI”; the magnitude/severity of the accident risk is not that relevant to this argument.
If you’re arguing against that, I’m still not sure what your counter-argument is. To me, the argument is: the upsides of nukes are the ability to take over the world (militarily) and to defend against such attempts. The downsides include risks of local and global catastrophe. People raced to develop nukes because they judged the upsides to be greater than the downsides, in part because they’re not altruists and longtermists. It seems like people will develop potentially unsafe AI for analogous reasons: the upsides include the ability to take over the world (militarily or economically) and to defend against such attempts, and the downsides include risks of local and global catastrophe, and people will likely race to develop AI because they judge the upsides to be greater than the downsides, in part because they’re not altruists and longtermists.
Where do you see this analogy breaking down?
I’m more sympathetic to this argument (which is a claim about what might happen in the future, as opposed to what is happening now, which is the analogy I usually encounter, though possibly not on LessWrong). I still think the analogy breaks down, though in different ways:
There is a strong norm of openness in AI research (though that might be changing). (Though perhaps this was the case with nuclear physics too.)
There is a strong anti-government / anti-military ethic in the AI research community. I’m not sure what the nuclear analog is, but I’m guessing it was neutral or pro-government/military.
Governments are staying a mile away from AGI; their interest in AI is in narrow AI’s applications. Narrow AI applications are diverse, and many can be done by a huge number of people. In contrast, nukes are a single technology, governments were interested in them, and only a few people could plausibly build them. (This is relevant if you think a ton of narrow AI could be used to take over the world economically.)
OpenAI / DeepMind are not adversarial towards each other. In contrast, US / Germany were definitely adversarial.
Assuming you agree that people are already pushing too hard for progress in AGI capability (relative to what’s ideal from a longtermist perspective), I think the current motivations for that are mostly things like money, prestige, scientific curiosity, wanting to make the world a better place (in a misguided/shorttermist way), etc., and not so much wanting to take over the world or to defend against such attempts. This seems likely to persist in the near future, but my concern is that if AGI research gets sufficiently close to fruition, governments will inevitably get involved and start pushing it even harder due to national security considerations. (Recall that Manhattan Project started 8 years before detonation of the first nuke.) Your argument seems more about what’s happening now, and does not really address this concern.
I’m uncertain, given the potential for AGI to be used to reduce other x-risks. (I don’t have strong opinions on how large other x-risks are and how much potential there is for AGI to differentially help.) But I’m happy to accept this as a premise.
I think what’s happening now is a good guide into what will happen in the future, at least on short timelines. If AGI is >100 years away, then sure, a lot will change and current facts are relatively unimportant. If it’s < 20 years away, then current facts seem very relevant. I usually focus on the shorter timelines.
For min(20 years, time till AGI), for each individual trend I identified, I’d weakly predict that trend will continue (except perhaps openness, because that’s already changing).
It wasn’t meant as a reply to a particular thing—mainly I’m flagging this as an AI-risk analogy I like.
On that theme, one thing “we don’t know if the nukes will ignite the atmosphere” has in common with AI-risk is that the risk is from reaching new configurations (e.g. temperatures of the sort you get out of a nuclear bomb inside the Earth’s atmosphere) that we don’t have experience with. Which is an entirely different question than “what happens with the nukes after we don’t ignite the atmosphere in a test explosion”.
I like thinking about coordination from this viewpoint.