And then what social system is left? A second wave of wars would be inevitable, and inevitably nuclear.
Ah, yes, societal collapse. Where was the societal collapse in Europe during WWI (which unluckily coincided with a flu pandemic that killed a lot of society’s most productive members, namely people in their 20s) or WWII? Where was the societal collapse in China during its civil war (the 20th Century’s third most deadly war) and then to top it off, during the civil war, Japan invaded China?
When you were writing this—let us be specific: when you were writing, “Despite fears about the alignment of interests between AI and Humanity, in reality what we know for sure is that the most intractable problem is the alignment among humans, and that [the] problem with nuclear weapons is also existential,” did you know that “existential risk” is usually used to mean risk of the human population becoming zero? I.e., literally no human left alive whatsoever? If so, you haven’t explained how the nuclear war would bring that about.
Suppose a nuclear war kills 80% of the human population (which I consider barely possible, but at the extreme tail of the distribution of outcomes of a nuclear war, entailing some vulnerability in human civilization that I am probably currently completely unaware of). What is the mechanism by which all of the remaining 20% die? If for example there is widespread societal collapse (which again I consider very unlikely) how would that bring about the death of literally everyone? Why wouldn’t for example some people survive as hunters, gatherers and small-scale farmers?
I think that there is a lot of uncertainty over the effects of EMP on electronics because electronics has changed drastically since the end of the Cold War, and when the Cold War ended, the number of very competent and committed people studying nuclear war decreased drastically. But even with this uncertainty, I think we can say that your “the majority of electronics destroyed” is very unlikely: the wikipedia page on nuclear EMP asserts that most vehicles and cell phones would survive nuclear EMP. The power grid I will concede will be damaged by EMP, but not as damaged as it will be by the other effects of the nuclear detonations (fires, overpressure strong enough to knock down most of the buildings). Again, how does that bring the human population of the world to zero? If the entire power grid became non-operational and stayed that way for 6 months or 3 years, what prevents the survivors of the nuclear war from temporarily adopting a social organization that feeds everyone and performs a few other socially-necessary functions until the electrical grid is restored? Or suppose that is unachievable, and only 50% of the survivors get enough food to survive (unlikely because for example the US has right now about 3 years worth of food mostly stored in grain elevators and intended to be fed to cows and pigs and such, but which could be diverted in an emergency for human use): how does that prevent the survivors of the starvation from painfully rebuilding post-industrial civilization?
I think that our civilization (and my country, the US) should drastically increase its efforts to prevent and to prepare for nuclear war. Its just the notion of turning to AGI research of all things to do so that I disagree with.
We would not be destroyed in the first large nuclear war (while effects of radioactivity in the food chains in my view are under researched). But not a single open society would survive.
A world of malthusian masses, and a military aristocracy desperately trying to keep as much firepower as possible is the natural post nuclear war outcome. Then, history happens again, and in 2000 years luckily we are back into deciding if we allow AGI to be developed. What is the point?
The baseline is that our governance systems are completely unaccurate for nuclear weapons. Even in this fortunate age of hegemonic republics.
We need to solve the human alignement problem. Do you have any better suggestion than AGI?
I find it frustrating to correspond with you. You have become attached to an argument for what we should do. To support this argument, you send out many “soldiers”: nuclear war is a potent existential risk; electromagnetic pulse would destroy most electronic devices; not a single open society would survive a nuclear war; nuclear war inevitably leads to more nuclear war; nuclear war will cause widespread societal collapse. And now we have a new soldier, namely, the effect of radiation on the food chain.
Each of these soldiers seems plausible if one’s epistemology consists mostly in noticing how often something is repeated in the press and online. But I haven’t seen a single attempt by you to support any of these assertions / soldiers. When I say that Wikipedia says that most vehicles and cellphones would continue to operate after the electromagnetic pulses of a nuclear war, you ignore that. I offer you an opening to explain why you believe that nuclear war will lead to societal collapse whereas WWI, WWII and the Chinese civil war did not; you decline to engage on that. I still do not know whether you accept the conventional definition of “existential risk” (even after I asked you a direct question): when you wrote that nuclear war is a potent existential risk, maybe you thought that the possibility that half of the human population might die constitutes an existential risk. I.e., maybe you have been using an unconventional definition of existential risk. Your readers (including me) still do not know.
If I continue corresponding with you, I expect you would send out a few more soldiers, but it take me a lot more work to explain why a soldier does not in fact support your argument than it takes you to find the next soldier and to send it out.
Have you ever tried to learn about the effects of radiation on the food chain, e.g., by typing the phrase into a search engine and spending 5 minutes (as measured by an actual clock or timer) looking at the results? Science knows much about the subject. The radiation from an accident at a nuclear power plant is very different from the radiation from a nuclear weapon, so you’d have to be careful not to generalize from the first case to the second. (A much higher fraction of the radiation in the first case comes from long-half-life isotopes.)
“Commercial computer equipment is particularly vulnerable to EMP effects. Computers used in data processing systems, communications systems, displays, industrial control applications, including road and rail signaling, and those embedded in military equipment, such as signal processors, electronic flight controls and digital engine control systems, are all potentially vulnerable to the EMP effect. Other electronic devices and electrical equipment may also be destroyed by the EMP effect. Telecommunications equipment can be highly vulnerable and receivers of all varieties are particularly sensitive to EMP. Therefore radar and electronic warfare equipment, satellite, microwave, UHF, VHF, HF and low band communications equipment and television equipment are all potentially vulnerable to the EMP effect. Cars with electronic ignition systems/ and ignition chips are also vulnerable.”
not a single open society would survive a nuclear war; nuclear war inevitably leads to more nuclear war; nuclear war will cause widespread societal collapse
:-) After total economic disruption, no electricity nor electronics, and a few billion deaths… its like the parachute randomized trial. Too obvious to be argued.
Now let´s compare with peer reviewed literature on AGI:
Ah, yes, societal collapse. Where was the societal collapse in Europe during WWI (which unluckily coincided with a flu pandemic that killed a lot of society’s most productive members, namely people in their 20s) or WWII? Where was the societal collapse in China during its civil war (the 20th Century’s third most deadly war) and then to top it off, during the civil war, Japan invaded China?
When you were writing this—let us be specific: when you were writing, “Despite fears about the alignment of interests between AI and Humanity, in reality what we know for sure is that the most intractable problem is the alignment among humans, and that [the] problem with nuclear weapons is also existential,” did you know that “existential risk” is usually used to mean risk of the human population becoming zero? I.e., literally no human left alive whatsoever? If so, you haven’t explained how the nuclear war would bring that about.
Suppose a nuclear war kills 80% of the human population (which I consider barely possible, but at the extreme tail of the distribution of outcomes of a nuclear war, entailing some vulnerability in human civilization that I am probably currently completely unaware of). What is the mechanism by which all of the remaining 20% die? If for example there is widespread societal collapse (which again I consider very unlikely) how would that bring about the death of literally everyone? Why wouldn’t for example some people survive as hunters, gatherers and small-scale farmers?
I think that there is a lot of uncertainty over the effects of EMP on electronics because electronics has changed drastically since the end of the Cold War, and when the Cold War ended, the number of very competent and committed people studying nuclear war decreased drastically. But even with this uncertainty, I think we can say that your “the majority of electronics destroyed” is very unlikely: the wikipedia page on nuclear EMP asserts that most vehicles and cell phones would survive nuclear EMP. The power grid I will concede will be damaged by EMP, but not as damaged as it will be by the other effects of the nuclear detonations (fires, overpressure strong enough to knock down most of the buildings). Again, how does that bring the human population of the world to zero? If the entire power grid became non-operational and stayed that way for 6 months or 3 years, what prevents the survivors of the nuclear war from temporarily adopting a social organization that feeds everyone and performs a few other socially-necessary functions until the electrical grid is restored? Or suppose that is unachievable, and only 50% of the survivors get enough food to survive (unlikely because for example the US has right now about 3 years worth of food mostly stored in grain elevators and intended to be fed to cows and pigs and such, but which could be diverted in an emergency for human use): how does that prevent the survivors of the starvation from painfully rebuilding post-industrial civilization?
I think that our civilization (and my country, the US) should drastically increase its efforts to prevent and to prepare for nuclear war. Its just the notion of turning to AGI research of all things to do so that I disagree with.
We would not be destroyed in the first large nuclear war (while effects of radioactivity in the food chains in my view are under researched). But not a single open society would survive.
A world of malthusian masses, and a military aristocracy desperately trying to keep as much firepower as possible is the natural post nuclear war outcome. Then, history happens again, and in 2000 years luckily we are back into deciding if we allow AGI to be developed. What is the point?
The baseline is that our governance systems are completely unaccurate for nuclear weapons. Even in this fortunate age of hegemonic republics.
We need to solve the human alignement problem. Do you have any better suggestion than AGI?
I find it frustrating to correspond with you. You have become attached to an argument for what we should do. To support this argument, you send out many “soldiers”: nuclear war is a potent existential risk; electromagnetic pulse would destroy most electronic devices; not a single open society would survive a nuclear war; nuclear war inevitably leads to more nuclear war; nuclear war will cause widespread societal collapse. And now we have a new soldier, namely, the effect of radiation on the food chain.
Each of these soldiers seems plausible if one’s epistemology consists mostly in noticing how often something is repeated in the press and online. But I haven’t seen a single attempt by you to support any of these assertions / soldiers. When I say that Wikipedia says that most vehicles and cellphones would continue to operate after the electromagnetic pulses of a nuclear war, you ignore that. I offer you an opening to explain why you believe that nuclear war will lead to societal collapse whereas WWI, WWII and the Chinese civil war did not; you decline to engage on that. I still do not know whether you accept the conventional definition of “existential risk” (even after I asked you a direct question): when you wrote that nuclear war is a potent existential risk, maybe you thought that the possibility that half of the human population might die constitutes an existential risk. I.e., maybe you have been using an unconventional definition of existential risk. Your readers (including me) still do not know.
If I continue corresponding with you, I expect you would send out a few more soldiers, but it take me a lot more work to explain why a soldier does not in fact support your argument than it takes you to find the next soldier and to send it out.
Have you ever tried to learn about the effects of radiation on the food chain, e.g., by typing the phrase into a search engine and spending 5 minutes (as measured by an actual clock or timer) looking at the results? Science knows much about the subject. The radiation from an accident at a nuclear power plant is very different from the radiation from a nuclear weapon, so you’d have to be careful not to generalize from the first case to the second. (A much higher fraction of the radiation in the first case comes from long-half-life isotopes.)
Nuclear winter:
https://climate.envsci.rutgers.edu/pdf/WiresClimateChangeNW.pdf
Electromagnetic pulse would destroy most electronic devices:
https://doh.wa.gov/sites/default/files/legacy/Documents/Pubs/320-090_elecpuls_fs.pdf
“Commercial computer equipment is particularly vulnerable to EMP effects. Computers used in data processing systems, communications systems, displays, industrial control applications, including road and rail signaling, and those embedded in military equipment, such as signal processors, electronic flight controls and digital engine control systems, are all potentially vulnerable to the EMP effect. Other electronic devices and electrical equipment may also be destroyed by the EMP effect. Telecommunications equipment can be highly vulnerable and receivers of all varieties are particularly sensitive to EMP. Therefore radar and electronic warfare equipment, satellite, microwave, UHF, VHF, HF and low band communications equipment and television equipment are all potentially vulnerable to the EMP effect. Cars with electronic ignition systems/ and ignition chips are also vulnerable.”
not a single open society would survive a nuclear war; nuclear war inevitably leads to more nuclear war; nuclear war will cause widespread societal collapse
Do you expect a paper? I have this one:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/
:-) After total economic disruption, no electricity nor electronics, and a few billion deaths… its like the parachute randomized trial. Too obvious to be argued.
Now let´s compare with peer reviewed literature on AGI:
https://marginalrevolution.com/marginalrevolution/2023/04/from-the-comments-on-ai-safety.html
“The only peer-reviewed paper making the case for AI risk that I know of is: https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064. Though note that my paper (the second you linked) is currently under review at a top ML conference.”