‘Nuclear power’ seems to me like a weird example because we selectively halted the development of productive use of nuclear power while having comparatively little standing in the way of development of destructive use of nuclear power. If a similar story holds, then we’ll still see militarily relevant AIs (deliberately doing adversarial planning of the sort that could lead to human extinction) while not getting many of the benefits along the way.
That… doesn’t seem like much of a coordination success story, to me.
Isn’t NPT a canonical example in international relations of coordination success? In the 60s people thought many states would acquire nukes in the next few decades, but a treaty essentially stopped new states from acquiring nukes and set up a structure for helping states use nuclear power non-militarily. (Then many states gradually, individually, domestically chose not to pursue nuclear power much, for reasons specific to nuclear power.)
Yes, because the standards for success for nuclear are much lower than they are for AI. Not only did 5 states acquire weapons before the treaty was signed, around four have acquired them since, and this didn’t stop the arms race accumulation of thousands of weapons. This turned out to be enough (so far).
In worlds where nukes ignite the atmosphere the first time you use them, there would have been a different standard of coordination necessary to count as ‘success’. (Or in worlds where we counted the non-signatory states, many of which have nuclear weapons, as failures.)
The point here is that the NPT is insufficient. With the alignment problem, it doesn’t matter much if one state has it or many individuals have it, it only matters if no one has it.
A better example is arguably biotech, and this only happened because WW2 torched the idea of human eugenics, thus indirectly slowing down biotech by preventing it’s funding.
The NPT framework, if it could be implemented, would be sufficient. The goal of the NPT is to enable countries to mutually verify that no additional country has acquired a nuclear weapon, while still enabling the spread of nuclear power to many more states. It has been pretty successful at this, with just a few new states gaining nuclear weapons over the last 50 years, whereas many more can enrich uranium/operate power plants.
It happens that the number of nuclear-armed countries at the NPT’s signing was nonzero, but if it had been 0, then the goal of the NPT would’ve been “no one anywhere can develop a nuclear weapon”.
A separate Q is “could we have implemented the NPT without Hiroshima, if scientists had strong evidence it would ignite the atmosphere?” People can have reasonable disagreements here; I think it’s lame not to try.
A separate Q is “could we have implemented the NPT without Hiroshima, if scientists had strong evidence it would ignite the atmosphere?” People can have reasonable disagreements here; I think it’s lame not to try.
The unfortunate answer is likely not, assuming the cold war happens like it did historically. Both sides were very much going to get nuclear weapons and escalate as soon as they were able to. You really need almost Alien Space Bats or random quantum events to prevent the historical outcome of several states getting nuclear weapons. Now w imagine those nuclear weapons were intelligent and misaligned, and the world probably goes up in flames. Not assuredly, but well over 50% probability per year.
‘Nuclear power’ seems to me like a weird example because we selectively halted the development of productive use of nuclear power while having comparatively little standing in the way of development of destructive use of nuclear power. If a similar story holds, then we’ll still see militarily relevant AIs (deliberately doing adversarial planning of the sort that could lead to human extinction) while not getting many of the benefits along the way.
That… doesn’t seem like much of a coordination success story, to me.
Isn’t NPT a canonical example in international relations of coordination success? In the 60s people thought many states would acquire nukes in the next few decades, but a treaty essentially stopped new states from acquiring nukes and set up a structure for helping states use nuclear power non-militarily. (Then many states gradually, individually, domestically chose not to pursue nuclear power much, for reasons specific to nuclear power.)
Yes, because the standards for success for nuclear are much lower than they are for AI. Not only did 5 states acquire weapons before the treaty was signed, around four have acquired them since, and this didn’t stop the arms race accumulation of thousands of weapons. This turned out to be enough (so far).
In worlds where nukes ignite the atmosphere the first time you use them, there would have been a different standard of coordination necessary to count as ‘success’. (Or in worlds where we counted the non-signatory states, many of which have nuclear weapons, as failures.)
The point here is that the NPT is insufficient. With the alignment problem, it doesn’t matter much if one state has it or many individuals have it, it only matters if no one has it.
A better example is arguably biotech, and this only happened because WW2 torched the idea of human eugenics, thus indirectly slowing down biotech by preventing it’s funding.
The NPT framework, if it could be implemented, would be sufficient. The goal of the NPT is to enable countries to mutually verify that no additional country has acquired a nuclear weapon, while still enabling the spread of nuclear power to many more states. It has been pretty successful at this, with just a few new states gaining nuclear weapons over the last 50 years, whereas many more can enrich uranium/operate power plants.
It happens that the number of nuclear-armed countries at the NPT’s signing was nonzero, but if it had been 0, then the goal of the NPT would’ve been “no one anywhere can develop a nuclear weapon”.
A separate Q is “could we have implemented the NPT without Hiroshima, if scientists had strong evidence it would ignite the atmosphere?” People can have reasonable disagreements here; I think it’s lame not to try.
The unfortunate answer is likely not, assuming the cold war happens like it did historically. Both sides were very much going to get nuclear weapons and escalate as soon as they were able to. You really need almost Alien Space Bats or random quantum events to prevent the historical outcome of several states getting nuclear weapons. Now w imagine those nuclear weapons were intelligent and misaligned, and the world probably goes up in flames. Not assuredly, but well over 50% probability per year.