Probably this is supposed to work like EY’s “nanobot swarm that melts all of the GPUs”.
I would like to see them state things a little more clearly than commentators having to guess ‘well probably it’s supposed to work sorta like this idk?‘, and I would also point out that even this (a strategy so far outside the Overton Window that people usually bring it up to mock EY as a lunatic) is not an easy cheap act if you actually sit down and think about it seriously in near mode as a concrete policy that, say, President Trump has to order, rather than ‘entertaining thought experiment far mode with actual humans replaced by hypercompetent automatically-strategic archetypes’.
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them—an invasion that no major power would accept lying down and would likely trigger a major backlash; once you start riding that tiger, you’re never getting off of it. Such an act would make a mockery of 103 years of CCP history and propaganda, and undermine every thing they have claimed to succeed at and explode ‘the China Dream’. (Just imagine if the Chinese did that to the USA? ‘Pearl Harbor’ or ‘Sputnik’ or ‘9/11’ might scarcely begin to cover how Americans would react.) And if such a strategy were on the table, it would likely have been preceded by explicit statements by sovereign nations that such actions would be considered equivalent to invasions or nuclear strikes and justifying response in kind. (Like, as it happens, has been a major focus of Xi’s military investments in order to more credibly threaten the US over actions elsewhere.)
To believe that the same group that pulled of Stuxnet
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
Stuxnet was designed to attack as few targets as possible and conceal itself thoroughly, and had no casualties, but it was still a major enterprise for the USA & Israel to authorize, going straight to the top with personal involvement from Bush & Obama themselves, at times seriously considering killing the entire effort (which the US continues to not acknowledge all these years later). Further, Stuxnet was not a decisive advantage, and the USA and Israel did nothing thanks to Stuxnet-caused delays which resulted in a permanent resolution to Iran and nukes: they did not invade, they did not permanently hack all Iranian nuclear programs and rendered work futile, they did not end the Iranian nuclear program, they did not any of that—and Iran continued low-key pursuing nukes right up to the present day. The only reason Iran doesn’t have nukes right now is not because it lacks a breakout capacity or was unable to do it long before if it had made that the #1 priority, but because it doesn’t want to enough. Not because of Stuxnet.
(It did, however, succeed in making them even angrier and paranoid and more embittered against the USA & Israel, and contributing to deterioration in relations and difficulties in the negotiations for a nuclear deal which were the closest any strategy has come to stopping Iran nuclearizing… It has also been criticized for inaugurating a new age of nation-state malware, so one might also ask the planners of “Olympic Games” what their plan was to ‘bury the body’ once their malware succeeded and was inevitably eventually discovered.)
It’s also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve.
Nukes were a hyper-exponential curve too. Large high-explosives mining, fire storms, conventional explosives like the Mother of All Bombs… IIRC AI Impacts has a page showing the increase in yield over time, and Hiroshima, being such a small nuke, is not as much of a “zero-to-one event” as one might think. Just a very sharp curve, exacerbated by additional developments like missiles and increases in yields, which can look zero-to-one if you looked away for a few years and had a low anchoring point.
I would like to see them state things a little more clearly than commentators having to guess ‘well probably it’s supposed to work sorta like this idk?’
Meh. I want the national security establishment to act like a national security establishment. I admit it is frustratingly opaque from the outside, but that does not mean I want more transparency at the cost of it being worse. Tactical Surprise and Strategic Ambiguity are real things with real benefits.
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
I think both can be true true: Stuxnet did not stop the Iranian nuclear program and if there was a “destroy all Chinese long-range weapons and High Performance Computing clusters” NATSEC would pound that button.
Is your argument that a 1-year head start on AGI is not enough to build such a button, or do you really think it wouldn’t be pressed?
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them—an invasion that no major power would accept lying down and would likely trigger a major backlash
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting. A very telling current example being the current Iranian non-response to Israel’s actions against Hamas/Hezbollah.
Nukes were a hyper-exponential curve too.
While this is a clever play on words, it is not a good argument. There are good reasons to expect AGI to affect the offense-defense balance in ways that are fundamentally different from nuclear weapons.
Tactical Surprise and Strategic Ambiguity are real things with real benefits.
And would imply that were one a serious thinker and proposing an arms race, one would not be talking about the arms race publicly. (By the way, I am told there are at least 5 different Chinese translations of “Situational Awareness” in circulation now.)
So, there is a dilemma: they are doing this poorly, either way. If you need to discuss the arms race in public, say to try to solve a coordination problem, you should explain what the exit plan is rather than uttering vague verbiage like “robust military advantage” (even if that puffery is apparently adequate for some readers); and if you cannot make a convincing public case, then you shouldn’t be arguing about it in public at all. Einstein didn’t write a half-assed NYT op-ed about how vague ‘advances in science’ might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
I think both can be true
Maybe, but then your example doesn’t prove it, if you are now conceding that Stuxnet is not a decisive advantage after all. If it was not, then NATSEC willingness to, hesitantly, push the Suxnet button is not relevant. And if it was, then the outcome also refutes you: they pushed the button, and it didn’t work. You chose a bad example for your claims.
and if there was a “destroy all Chinese long-range weapons and High Performance Computing clusters” NATSEC would pound that button.
Note what you just did there. You specified a precise strategy: “nanobot swarm that melts all of the GPUs”. I pointed out just some of the many problems with it, which are why one would almost certainly choose to not execute it, and you have silently amended it to “nanobot swarm that melts all of the GPUs and all Chinese long-range weapons”. What other issues might there be with this new ad hoced strategy...?
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting.
...for example, let me just note this: “destroyed long-range military assets can be replaced”{{citation needed}}.
While this is a clever play on words, it is not a good argument
Then why did you bring it up in the first place as a thing which distinguished nukes from AGI, when it did not, and your response to that rebuttal is to dismiss ‘hyper-exponential’ as mere word-play?
Einstein didn’t write a half-assed NYT op-ed about how vague ‘advances in science’ might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
Strongly agree.
What other issues might there be with this new ad hoced strategy...?
I am not a China Hawk. I do not speak for the China Hawks. I 100% concede your argument that these conversations should be taking place in a room that neither you our I are in right now.
I would like to see them state things a little more clearly than commentators having to guess ‘well probably it’s supposed to work sorta like this idk?‘, and I would also point out that even this (a strategy so far outside the Overton Window that people usually bring it up to mock EY as a lunatic) is not an easy cheap act if you actually sit down and think about it seriously in near mode as a concrete policy that, say, President Trump has to order, rather than ‘entertaining thought experiment far mode with actual humans replaced by hypercompetent automatically-strategic archetypes’.
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them—an invasion that no major power would accept lying down and would likely trigger a major backlash; once you start riding that tiger, you’re never getting off of it. Such an act would make a mockery of 103 years of CCP history and propaganda, and undermine every thing they have claimed to succeed at and explode ‘the China Dream’. (Just imagine if the Chinese did that to the USA? ‘Pearl Harbor’ or ‘Sputnik’ or ‘9/11’ might scarcely begin to cover how Americans would react.) And if such a strategy were on the table, it would likely have been preceded by explicit statements by sovereign nations that such actions would be considered equivalent to invasions or nuclear strikes and justifying response in kind. (Like, as it happens, has been a major focus of Xi’s military investments in order to more credibly threaten the US over actions elsewhere.)
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
Stuxnet was designed to attack as few targets as possible and conceal itself thoroughly, and had no casualties, but it was still a major enterprise for the USA & Israel to authorize, going straight to the top with personal involvement from Bush & Obama themselves, at times seriously considering killing the entire effort (which the US continues to not acknowledge all these years later). Further, Stuxnet was not a decisive advantage, and the USA and Israel did nothing thanks to Stuxnet-caused delays which resulted in a permanent resolution to Iran and nukes: they did not invade, they did not permanently hack all Iranian nuclear programs and rendered work futile, they did not end the Iranian nuclear program, they did not any of that—and Iran continued low-key pursuing nukes right up to the present day. The only reason Iran doesn’t have nukes right now is not because it lacks a breakout capacity or was unable to do it long before if it had made that the #1 priority, but because it doesn’t want to enough. Not because of Stuxnet.
(It did, however, succeed in making them even angrier and paranoid and more embittered against the USA & Israel, and contributing to deterioration in relations and difficulties in the negotiations for a nuclear deal which were the closest any strategy has come to stopping Iran nuclearizing… It has also been criticized for inaugurating a new age of nation-state malware, so one might also ask the planners of “Olympic Games” what their plan was to ‘bury the body’ once their malware succeeded and was inevitably eventually discovered.)
Nukes were a hyper-exponential curve too. Large high-explosives mining, fire storms, conventional explosives like the Mother of All Bombs… IIRC AI Impacts has a page showing the increase in yield over time, and Hiroshima, being such a small nuke, is not as much of a “zero-to-one event” as one might think. Just a very sharp curve, exacerbated by additional developments like missiles and increases in yields, which can look zero-to-one if you looked away for a few years and had a low anchoring point.
Meh. I want the national security establishment to act like a national security establishment. I admit it is frustratingly opaque from the outside, but that does not mean I want more transparency at the cost of it being worse. Tactical Surprise and Strategic Ambiguity are real things with real benefits.
I think both can be true true: Stuxnet did not stop the Iranian nuclear program and if there was a “destroy all Chinese long-range weapons and High Performance Computing clusters” NATSEC would pound that button.
Is your argument that a 1-year head start on AGI is not enough to build such a button, or do you really think it wouldn’t be pressed?
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting. A very telling current example being the current Iranian non-response to Israel’s actions against Hamas/Hezbollah.
While this is a clever play on words, it is not a good argument. There are good reasons to expect AGI to affect the offense-defense balance in ways that are fundamentally different from nuclear weapons.
And would imply that were one a serious thinker and proposing an arms race, one would not be talking about the arms race publicly. (By the way, I am told there are at least 5 different Chinese translations of “Situational Awareness” in circulation now.)
So, there is a dilemma: they are doing this poorly, either way. If you need to discuss the arms race in public, say to try to solve a coordination problem, you should explain what the exit plan is rather than uttering vague verbiage like “robust military advantage” (even if that puffery is apparently adequate for some readers); and if you cannot make a convincing public case, then you shouldn’t be arguing about it in public at all. Einstein didn’t write a half-assed NYT op-ed about how vague ‘advances in science’ might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
Maybe, but then your example doesn’t prove it, if you are now conceding that Stuxnet is not a decisive advantage after all. If it was not, then NATSEC willingness to, hesitantly, push the Suxnet button is not relevant. And if it was, then the outcome also refutes you: they pushed the button, and it didn’t work. You chose a bad example for your claims.
Note what you just did there. You specified a precise strategy: “nanobot swarm that melts all of the GPUs”. I pointed out just some of the many problems with it, which are why one would almost certainly choose to not execute it, and you have silently amended it to “nanobot swarm that melts all of the GPUs and all Chinese long-range weapons”. What other issues might there be with this new ad hoced strategy...?
...for example, let me just note this: “destroyed long-range military assets can be replaced”{{citation needed}}.
Then why did you bring it up in the first place as a thing which distinguished nukes from AGI, when it did not, and your response to that rebuttal is to dismiss ‘hyper-exponential’ as mere word-play?
Strongly agree.
I am not a China Hawk. I do not speak for the China Hawks. I 100% concede your argument that these conversations should be taking place in a room that neither you our I are in right now.