This question is a perfect mirror of the brain-dead “how is AGI going to kill us?” question. I could easily make a list of 100 things you might do if you had AGI supremacy and wanted to suppress the development of AGI in China. But the whole pointof AGI is that it will be smarter than me, so anything I put on the list would be redundant.
Missing the point. This is not about being too stupid to think of >0 strategies, this is about being able & willing to execute strategies.
I too can think of 100 things, and I listed several diverse ways of responding and threw in a historical parallel just in case that wasn’t clear after several paragraphs of discussing the problem with not having a viable strategy you can execute. Smartness is not the limit here: we are already smart enough to come up with strategies which could achieve the goal. All of those could potentially work. But none of them seem realistically on the table as something that the USA as it currently exists would be willing to commit to and see through to completion, and you will note that few critics—and no one serious—is responding something like, “oh sure, all part of the plan already, see our white paper laying out the roadmap: after we win, we would then order the AGIs to hack the planet and ensure our perpetual hegemony; that is indeed the exit plan. We botched it last time with nukes and stood by and let everyone else get nukes, but we’ll follow through this time.”
There is no difference between “won’t execute a strategy” and “can’t execute a strategy”: they are the same thing. The point is that a strategy (like a threat) has to be executable or else it’s not an actual strategy. And acting as if you can execute a strategy that you won’t can lead you to take terrible decisions. You are like the cat who thinks before climbing a tree: “obviously, I will just climb back down”, and who then proceeds climb up and to not climb back down but mew piteously. Well, maybe you shouldn’t’ve climbed up in the first place then...?
(“arms race bros will srsly launch a global arms race by saying they’ll use the decisive advantage from winning the arms race to conquer the world, and then will not conquer the world”)
Okay, this at least helps me better understand your position. Maybe you should have opened with “China Hawks won’t do the thing they’ve explicitly and repeatedly said they are going to do”
No, my problem with the hawks, as far as this criticism goes, is that they aren’t repeatedly and explicitly saying what they will do. (They also won’t do it, whatever ‘it’ is, even if they say they will; but we haven’t even gotten that far yet.) They are continually shying away from cashing out any of their post-AGI plans, likely because they look at the actual strategies that could be executed and realize that execution is in serious doubt and so that undermines their entire paradigm. (“We will be greeted as liberators” and “we don’t do nation-building” come to mind.)
Your quoted uses are a case in point of the substitution of rhetoric for substance. ‘Robust military superiority’ is not a decisive advantage in this sense, and is not ‘conquering the world’ or executing any of the strategies I mentioned; and in fact, this sort of vague bait-and-switch handwaving rhetoric, which is either wrong or deceptive about what they mean, is much of what I am criticizing: Oh, you have ‘robust military superiority’? That’s nice. But how does it actually stop Xi from getting AGI? Be concrete. How, exactly, do you go from eg. ‘the USA has some cool new bombs and nanotech thanks to running hundreds of thousands of Von Neumann AGI instances’ to ‘China [and every other rival country] has no AGI program and will not for the foreseeable future’?
The USA, for example, has always had ‘robust military superiority’ over many countries it desired to not get nukes, and yet, which did get nukes. (If you don’t like the early Cold War USSR example, then consider, say, North Korea pre-2006. The USA has always had ‘robust military superiority’ over the DPRK, and yet, here we are with Kim Jong Un having USA-range ICBMs and nukes. Why? Because the USA has always looked at the cost of using that ‘robust military superiority’, which would entail the destruction of Seoul and possibly millions of deaths and the provoking of major geopolitical powers—such as a certain CCP—and decided it was not worth the candle, and blinked, and kicked the can down the road, and after about three decades of can-kicking, ran out of road. Because the DPRK made nukes its #1 priority, ahead of lesser priorities like ‘not starving to death’, and it turns out that it’s rather hard to compel a sovereign country—even an extremely impoverished, isolated, weak country suffering from regular famines—to not pursue its #1 priority. It’s a lot easier to dissuade it from its #100 priority or something. But from #1? Difficult. Very difficult.)
Indeed, the USA has long had ‘robust military superiority’ over almost every country in the world not named “China” or “Russia”, and yet, those other countries continue doing many things the USA doesn’t like.{{citation needed}} So having ‘robust military superiority’ is perhaps not all it’s cracked up to be...
All this statement means is that ‘you lose even if you win’: 1. You race to AGI, ‘win’, you gain ‘robust military superiority’ which means something like “the USA can conquer China or otherwise force it to credibly terminate all AGI-related activities, if it’s willing to start a AGI-powered world war which will kill tens of millions of Chinese and crash the global economy (in the best case scenario)”; 2. Xi launches the national emergency crash AGI program like a ‘two bombs, one satellite’ program as the top national priority, the USA threatens to use its ‘robust military superiority’ if that AGI program is not canceled and condescendingly offers table scraps like gimped APIs, Xi says “no ur mom, btw, I have lots of nukes and cities to spare for the sake of China’s future”… and then what? Answer: no world war starts, and the Chinese AGI program finishes on schedule as if that ‘robust military superiority’ never existed. (A threat both sides know will not be executed is no threat at all.) 3. ??? 4. Profit!
(“arms race bros will srsly launch a global arms race by saying they’ll use the robust military superiority from winning the arms race to stop rival AGI programs, and then will not stop rival AGI programs”)
Because the USA has always looked at the cost of using that ‘robust military superiority’, which would entail the destruction of Seoul and possibly millions of deaths and the provoking of major geopolitical powers—such as a certain CCP—and decided it was not worth the candle, and blinked, and kicked the can down the road, and after about three decades of can-kicking, ran out of road.
I can’t explicitly speak for the China Hawks (not being one myself), but I believe one of the working assumptions is that AGI will allow the “league of free nations” to disarm China without the messiness of millions of deaths. Probably this is supposed to work like EY’s “nanobot swarm that melts all of the GPUs”.
I agree that the details are a bit fuzzy, but from an external perspective “we don’t publicly discuss capabilities” and “there are no adults in the room” are indistinguishable. OpenAI openly admits the plan is “we’ll as the AGI what to do”. I suspect NATSEC’s position is more like “amateurs discuss tactics, experts discuss logistics” (i.e. securing decisive advantage is more important that planning out exactly how to melt the GPUs)
To believe that the same group that pulled of Stuxnet and this lack the imagination or will to use AGI enabled weapons strikes me as naive, however.
The USA, for example, has always had ‘robust military superiority’ over many countries it desired to not get nukes, and yet, which did get nukes.
It’s also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve. Theoretically it may be possible to always stay far-enough-ahead to have decisive advantage (unlike nukes where even a handful is enough to establish MAD).
Probably this is supposed to work like EY’s “nanobot swarm that melts all of the GPUs”.
I would like to see them state things a little more clearly than commentators having to guess ‘well probably it’s supposed to work sorta like this idk?‘, and I would also point out that even this (a strategy so far outside the Overton Window that people usually bring it up to mock EY as a lunatic) is not an easy cheap act if you actually sit down and think about it seriously in near mode as a concrete policy that, say, President Trump has to order, rather than ‘entertaining thought experiment far mode with actual humans replaced by hypercompetent automatically-strategic archetypes’.
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them—an invasion that no major power would accept lying down and would likely trigger a major backlash; once you start riding that tiger, you’re never getting off of it. Such an act would make a mockery of 103 years of CCP history and propaganda, and undermine every thing they have claimed to succeed at and explode ‘the China Dream’. (Just imagine if the Chinese did that to the USA? ‘Pearl Harbor’ or ‘Sputnik’ or ‘9/11’ might scarcely begin to cover how Americans would react.) And if such a strategy were on the table, it would likely have been preceded by explicit statements by sovereign nations that such actions would be considered equivalent to invasions or nuclear strikes and justifying response in kind. (Like, as it happens, has been a major focus of Xi’s military investments in order to more credibly threaten the US over actions elsewhere.)
To believe that the same group that pulled of Stuxnet
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
Stuxnet was designed to attack as few targets as possible and conceal itself thoroughly, and had no casualties, but it was still a major enterprise for the USA & Israel to authorize, going straight to the top with personal involvement from Bush & Obama themselves, at times seriously considering killing the entire effort (which the US continues to not acknowledge all these years later). Further, Stuxnet was not a decisive advantage, and the USA and Israel did nothing thanks to Stuxnet-caused delays which resulted in a permanent resolution to Iran and nukes: they did not invade, they did not permanently hack all Iranian nuclear programs and rendered work futile, they did not end the Iranian nuclear program, they did not any of that—and Iran continued low-key pursuing nukes right up to the present day. The only reason Iran doesn’t have nukes right now is not because it lacks a breakout capacity or was unable to do it long before if it had made that the #1 priority, but because it doesn’t want to enough. Not because of Stuxnet.
(It did, however, succeed in making them even angrier and paranoid and more embittered against the USA & Israel, and contributing to deterioration in relations and difficulties in the negotiations for a nuclear deal which were the closest any strategy has come to stopping Iran nuclearizing… It has also been criticized for inaugurating a new age of nation-state malware, so one might also ask the planners of “Olympic Games” what their plan was to ‘bury the body’ once their malware succeeded and was inevitably eventually discovered.)
It’s also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve.
Nukes were a hyper-exponential curve too. Large high-explosives mining, fire storms, conventional explosives like the Mother of All Bombs… IIRC AI Impacts has a page showing the increase in yield over time, and Hiroshima, being such a small nuke, is not as much of a “zero-to-one event” as one might think. Just a very sharp curve, exacerbated by additional developments like missiles and increases in yields, which can look zero-to-one if you looked away for a few years and had a low anchoring point.
I would like to see them state things a little more clearly than commentators having to guess ‘well probably it’s supposed to work sorta like this idk?’
Meh. I want the national security establishment to act like a national security establishment. I admit it is frustratingly opaque from the outside, but that does not mean I want more transparency at the cost of it being worse. Tactical Surprise and Strategic Ambiguity are real things with real benefits.
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
I think both can be true true: Stuxnet did not stop the Iranian nuclear program and if there was a “destroy all Chinese long-range weapons and High Performance Computing clusters” NATSEC would pound that button.
Is your argument that a 1-year head start on AGI is not enough to build such a button, or do you really think it wouldn’t be pressed?
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them—an invasion that no major power would accept lying down and would likely trigger a major backlash
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting. A very telling current example being the current Iranian non-response to Israel’s actions against Hamas/Hezbollah.
Nukes were a hyper-exponential curve too.
While this is a clever play on words, it is not a good argument. There are good reasons to expect AGI to affect the offense-defense balance in ways that are fundamentally different from nuclear weapons.
Tactical Surprise and Strategic Ambiguity are real things with real benefits.
And would imply that were one a serious thinker and proposing an arms race, one would not be talking about the arms race publicly. (By the way, I am told there are at least 5 different Chinese translations of “Situational Awareness” in circulation now.)
So, there is a dilemma: they are doing this poorly, either way. If you need to discuss the arms race in public, say to try to solve a coordination problem, you should explain what the exit plan is rather than uttering vague verbiage like “robust military advantage” (even if that puffery is apparently adequate for some readers); and if you cannot make a convincing public case, then you shouldn’t be arguing about it in public at all. Einstein didn’t write a half-assed NYT op-ed about how vague ‘advances in science’ might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
I think both can be true
Maybe, but then your example doesn’t prove it, if you are now conceding that Stuxnet is not a decisive advantage after all. If it was not, then NATSEC willingness to, hesitantly, push the Suxnet button is not relevant. And if it was, then the outcome also refutes you: they pushed the button, and it didn’t work. You chose a bad example for your claims.
and if there was a “destroy all Chinese long-range weapons and High Performance Computing clusters” NATSEC would pound that button.
Note what you just did there. You specified a precise strategy: “nanobot swarm that melts all of the GPUs”. I pointed out just some of the many problems with it, which are why one would almost certainly choose to not execute it, and you have silently amended it to “nanobot swarm that melts all of the GPUs and all Chinese long-range weapons”. What other issues might there be with this new ad hoced strategy...?
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting.
...for example, let me just note this: “destroyed long-range military assets can be replaced”{{citation needed}}.
While this is a clever play on words, it is not a good argument
Then why did you bring it up in the first place as a thing which distinguished nukes from AGI, when it did not, and your response to that rebuttal is to dismiss ‘hyper-exponential’ as mere word-play?
Einstein didn’t write a half-assed NYT op-ed about how vague ‘advances in science’ might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
Strongly agree.
What other issues might there be with this new ad hoced strategy...?
I am not a China Hawk. I do not speak for the China Hawks. I 100% concede your argument that these conversations should be taking place in a room that neither you our I are in right now.
I find this argument highly compelling. I think it’s necessary to actually think through those 100 ways to prevent rivals from gaining AGI if you already have one. And to be realistic about the rate of progress that AGI. We will not immediately have unstoppable nanobots. To be safe, you’d need some way to not only stop the use of Chinese and Russian nukes, but reliably keep them disabled. To prevent massive bloodshed, you’d also probably need to do the same with conventional military assets—and probably without causing massive casualties.
Diplomatic solutions are probably going to be part of any realistic plan to use AGI to prevent rival AGI—but as you say they won’t be enough.
Nonproliferation efforts for nukes slowed down proliferation but didn’t stop it. AGI is different in that it will fairly quickly allow nearly universal surveillance—if you can stomach deploying it, and if you don’t trigger a nuclear exchange by deploying it.
The other possibly-important difference between this scenario and the history of nuclear proliferation is the presence of a smarter-than-human advisor who can say “no really human, if you fail to follow through, these will be the very likely results, and you won’t like them”.
I also hope that smarter-than-human advisor will say something like “look guys, you can all get vastly wealthier and longer-lived if you can just not freak out and fight each other”—and be so obviously right and convincing that humans will actually listen. The win-win solutions may just be compelling. I fully agree that no amount of sharing will prevent others from pursuing AGI—but generous sharing of technological benefits would reduce the priority of those efforts and the animosity when they’re thwarted.
Now is the time to think this through carefully, before the US commits to a race.
No, my problem with the hawks, as far as this criticism goes, is that they aren’t repeatedly and explicitly saying what they will do
One issue with “explicitly and repeatedly saying what they will do” is that it invites competition. Many of the things that China hawks might want to do would be outside the Overton window. As Eliezer describes in AGI ruin:
The example I usually give is “burn all GPUs”. This is not what I think you’d actually want to do with a powerful AGI—the nanomachines would need to operate in an incredibly complicated open environment to hunt down all the GPUs, and that would be needlessly difficult to align. However, all known pivotal acts are currently outside the Overton Window, and I expect them to stay there. So I picked an example where if anybody says “how dare you propose burning all GPUs?” I can say “Oh, well, I don’t actually advocate doing that; it’s just a mild overestimate for the rough power level of what you’d have to do, and the rough level of machine cognition required to do that, in order to prevent somebody else from destroying the world in six months or three years.”
This question is a perfect mirror of the brain-dead “how is AGI going to kill us?” question. I could easily make a list of 100 things you might do if you had AGI supremacy and wanted to suppress the development of AGI in China. But the whole point of AGI is that it will be smarter than me, so anything I put on the list would be redundant.
Missing the point. This is not about being too stupid to think of >0 strategies, this is about being able & willing to execute strategies.
I too can think of 100 things, and I listed several diverse ways of responding and threw in a historical parallel just in case that wasn’t clear after several paragraphs of discussing the problem with not having a viable strategy you can execute. Smartness is not the limit here: we are already smart enough to come up with strategies which could achieve the goal. All of those could potentially work. But none of them seem realistically on the table as something that the USA as it currently exists would be willing to commit to and see through to completion, and you will note that few critics—and no one serious—is responding something like, “oh sure, all part of the plan already, see our white paper laying out the roadmap: after we win, we would then order the AGIs to hack the planet and ensure our perpetual hegemony; that is indeed the exit plan. We botched it last time with nukes and stood by and let everyone else get nukes, but we’ll follow through this time.”
There is no difference between “won’t execute a strategy” and “can’t execute a strategy”: they are the same thing. The point is that a strategy (like a threat) has to be executable or else it’s not an actual strategy. And acting as if you can execute a strategy that you won’t can lead you to take terrible decisions. You are like the cat who thinks before climbing a tree: “obviously, I will just climb back down”, and who then proceeds climb up and to not climb back down but mew piteously. Well, maybe you shouldn’t’ve climbed up in the first place then...?
(“arms race bros will srsly launch a global arms race by saying they’ll use the decisive advantage from winning the arms race to conquer the world, and then will not conquer the world”)
Okay, this at least helps me better understand your position. Maybe you should have opened with “China Hawks won’t do the thing they’ve explicitly and repeatedly said they are going to do”
No, my problem with the hawks, as far as this criticism goes, is that they aren’t repeatedly and explicitly saying what they will do. (They also won’t do it, whatever ‘it’ is, even if they say they will; but we haven’t even gotten that far yet.) They are continually shying away from cashing out any of their post-AGI plans, likely because they look at the actual strategies that could be executed and realize that execution is in serious doubt and so that undermines their entire paradigm. (“We will be greeted as liberators” and “we don’t do nation-building” come to mind.)
Your quoted uses are a case in point of the substitution of rhetoric for substance. ‘Robust military superiority’ is not a decisive advantage in this sense, and is not ‘conquering the world’ or executing any of the strategies I mentioned; and in fact, this sort of vague bait-and-switch handwaving rhetoric, which is either wrong or deceptive about what they mean, is much of what I am criticizing: Oh, you have ‘robust military superiority’? That’s nice. But how does it actually stop Xi from getting AGI? Be concrete. How, exactly, do you go from eg. ‘the USA has some cool new bombs and nanotech thanks to running hundreds of thousands of Von Neumann AGI instances’ to ‘China [and every other rival country] has no AGI program and will not for the foreseeable future’?
The USA, for example, has always had ‘robust military superiority’ over many countries it desired to not get nukes, and yet, which did get nukes. (If you don’t like the early Cold War USSR example, then consider, say, North Korea pre-2006. The USA has always had ‘robust military superiority’ over the DPRK, and yet, here we are with Kim Jong Un having USA-range ICBMs and nukes. Why? Because the USA has always looked at the cost of using that ‘robust military superiority’, which would entail the destruction of Seoul and possibly millions of deaths and the provoking of major geopolitical powers—such as a certain CCP—and decided it was not worth the candle, and blinked, and kicked the can down the road, and after about three decades of can-kicking, ran out of road. Because the DPRK made nukes its #1 priority, ahead of lesser priorities like ‘not starving to death’, and it turns out that it’s rather hard to compel a sovereign country—even an extremely impoverished, isolated, weak country suffering from regular famines—to not pursue its #1 priority. It’s a lot easier to dissuade it from its #100 priority or something. But from #1? Difficult. Very difficult.)
Indeed, the USA has long had ‘robust military superiority’ over almost every country in the world not named “China” or “Russia”, and yet, those other countries continue doing many things the USA doesn’t like.{{citation needed}} So having ‘robust military superiority’ is perhaps not all it’s cracked up to be...
All this statement means is that ‘you lose even if you win’: 1. You race to AGI, ‘win’, you gain ‘robust military superiority’ which means something like “the USA can conquer China or otherwise force it to credibly terminate all AGI-related activities, if it’s willing to start a AGI-powered world war which will kill tens of millions of Chinese and crash the global economy (in the best case scenario)”; 2. Xi launches the national emergency crash AGI program like a ‘two bombs, one satellite’ program as the top national priority, the USA threatens to use its ‘robust military superiority’ if that AGI program is not canceled and condescendingly offers table scraps like gimped APIs, Xi says “no ur mom, btw, I have lots of nukes and cities to spare for the sake of China’s future”… and then what? Answer: no world war starts, and the Chinese AGI program finishes on schedule as if that ‘robust military superiority’ never existed. (A threat both sides know will not be executed is no threat at all.) 3. ??? 4. Profit!
(“arms race bros will srsly launch a global arms race by saying they’ll use the robust military superiority from winning the arms race to stop rival AGI programs, and then will not stop rival AGI programs”)
I can’t explicitly speak for the China Hawks (not being one myself), but I believe one of the working assumptions is that AGI will allow the “league of free nations” to disarm China without the messiness of millions of deaths. Probably this is supposed to work like EY’s “nanobot swarm that melts all of the GPUs”.
I agree that the details are a bit fuzzy, but from an external perspective “we don’t publicly discuss capabilities” and “there are no adults in the room” are indistinguishable. OpenAI openly admits the plan is “we’ll as the AGI what to do”. I suspect NATSEC’s position is more like “amateurs discuss tactics, experts discuss logistics” (i.e. securing decisive advantage is more important that planning out exactly how to melt the GPUs)
To believe that the same group that pulled of Stuxnet and this lack the imagination or will to use AGI enabled weapons strikes me as naive, however.
It’s also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve. Theoretically it may be possible to always stay far-enough-ahead to have decisive advantage (unlike nukes where even a handful is enough to establish MAD).
I would like to see them state things a little more clearly than commentators having to guess ‘well probably it’s supposed to work sorta like this idk?‘, and I would also point out that even this (a strategy so far outside the Overton Window that people usually bring it up to mock EY as a lunatic) is not an easy cheap act if you actually sit down and think about it seriously in near mode as a concrete policy that, say, President Trump has to order, rather than ‘entertaining thought experiment far mode with actual humans replaced by hypercompetent automatically-strategic archetypes’.
It is a major, overt act of war and utter alarming shameful humiliating existential loss of national sovereignty which crosses red lines so red that no one has even had to state them—an invasion that no major power would accept lying down and would likely trigger a major backlash; once you start riding that tiger, you’re never getting off of it. Such an act would make a mockery of 103 years of CCP history and propaganda, and undermine every thing they have claimed to succeed at and explode ‘the China Dream’. (Just imagine if the Chinese did that to the USA? ‘Pearl Harbor’ or ‘Sputnik’ or ‘9/11’ might scarcely begin to cover how Americans would react.) And if such a strategy were on the table, it would likely have been preceded by explicit statements by sovereign nations that such actions would be considered equivalent to invasions or nuclear strikes and justifying response in kind. (Like, as it happens, has been a major focus of Xi’s military investments in order to more credibly threaten the US over actions elsewhere.)
A great example, thank you for reminding me of it as an illustration of the futility of these weak measures which are the available strategies to execute.
Stuxnet was designed to attack as few targets as possible and conceal itself thoroughly, and had no casualties, but it was still a major enterprise for the USA & Israel to authorize, going straight to the top with personal involvement from Bush & Obama themselves, at times seriously considering killing the entire effort (which the US continues to not acknowledge all these years later). Further, Stuxnet was not a decisive advantage, and the USA and Israel did nothing thanks to Stuxnet-caused delays which resulted in a permanent resolution to Iran and nukes: they did not invade, they did not permanently hack all Iranian nuclear programs and rendered work futile, they did not end the Iranian nuclear program, they did not any of that—and Iran continued low-key pursuing nukes right up to the present day. The only reason Iran doesn’t have nukes right now is not because it lacks a breakout capacity or was unable to do it long before if it had made that the #1 priority, but because it doesn’t want to enough. Not because of Stuxnet.
(It did, however, succeed in making them even angrier and paranoid and more embittered against the USA & Israel, and contributing to deterioration in relations and difficulties in the negotiations for a nuclear deal which were the closest any strategy has come to stopping Iran nuclearizing… It has also been criticized for inaugurating a new age of nation-state malware, so one might also ask the planners of “Olympic Games” what their plan was to ‘bury the body’ once their malware succeeded and was inevitably eventually discovered.)
Nukes were a hyper-exponential curve too. Large high-explosives mining, fire storms, conventional explosives like the Mother of All Bombs… IIRC AI Impacts has a page showing the increase in yield over time, and Hiroshima, being such a small nuke, is not as much of a “zero-to-one event” as one might think. Just a very sharp curve, exacerbated by additional developments like missiles and increases in yields, which can look zero-to-one if you looked away for a few years and had a low anchoring point.
Meh. I want the national security establishment to act like a national security establishment. I admit it is frustratingly opaque from the outside, but that does not mean I want more transparency at the cost of it being worse. Tactical Surprise and Strategic Ambiguity are real things with real benefits.
I think both can be true true: Stuxnet did not stop the Iranian nuclear program and if there was a “destroy all Chinese long-range weapons and High Performance Computing clusters” NATSEC would pound that button.
Is your argument that a 1-year head start on AGI is not enough to build such a button, or do you really think it wouldn’t be pressed?
The game theory implications of China waking up to finding all of their long-range military assets and GPUs have been destroyed are not what you are suggesting. A very telling current example being the current Iranian non-response to Israel’s actions against Hamas/Hezbollah.
While this is a clever play on words, it is not a good argument. There are good reasons to expect AGI to affect the offense-defense balance in ways that are fundamentally different from nuclear weapons.
And would imply that were one a serious thinker and proposing an arms race, one would not be talking about the arms race publicly. (By the way, I am told there are at least 5 different Chinese translations of “Situational Awareness” in circulation now.)
So, there is a dilemma: they are doing this poorly, either way. If you need to discuss the arms race in public, say to try to solve a coordination problem, you should explain what the exit plan is rather than uttering vague verbiage like “robust military advantage” (even if that puffery is apparently adequate for some readers); and if you cannot make a convincing public case, then you shouldn’t be arguing about it in public at all. Einstein didn’t write a half-assed NYT op-ed about how vague ‘advances in science’ might soon lead to new weapons of war and the USA should do something about that; he wrote a secret letter hand-delivered & pitched to President Roosevelt by a trusted advisor.
Maybe, but then your example doesn’t prove it, if you are now conceding that Stuxnet is not a decisive advantage after all. If it was not, then NATSEC willingness to, hesitantly, push the Suxnet button is not relevant. And if it was, then the outcome also refutes you: they pushed the button, and it didn’t work. You chose a bad example for your claims.
Note what you just did there. You specified a precise strategy: “nanobot swarm that melts all of the GPUs”. I pointed out just some of the many problems with it, which are why one would almost certainly choose to not execute it, and you have silently amended it to “nanobot swarm that melts all of the GPUs and all Chinese long-range weapons”. What other issues might there be with this new ad hoced strategy...?
...for example, let me just note this: “destroyed long-range military assets can be replaced”{{citation needed}}.
Then why did you bring it up in the first place as a thing which distinguished nukes from AGI, when it did not, and your response to that rebuttal is to dismiss ‘hyper-exponential’ as mere word-play?
Strongly agree.
I am not a China Hawk. I do not speak for the China Hawks. I 100% concede your argument that these conversations should be taking place in a room that neither you our I are in right now.
I find this argument highly compelling. I think it’s necessary to actually think through those 100 ways to prevent rivals from gaining AGI if you already have one. And to be realistic about the rate of progress that AGI. We will not immediately have unstoppable nanobots. To be safe, you’d need some way to not only stop the use of Chinese and Russian nukes, but reliably keep them disabled. To prevent massive bloodshed, you’d also probably need to do the same with conventional military assets—and probably without causing massive casualties.
Diplomatic solutions are probably going to be part of any realistic plan to use AGI to prevent rival AGI—but as you say they won’t be enough.
Nonproliferation efforts for nukes slowed down proliferation but didn’t stop it. AGI is different in that it will fairly quickly allow nearly universal surveillance—if you can stomach deploying it, and if you don’t trigger a nuclear exchange by deploying it.
The other possibly-important difference between this scenario and the history of nuclear proliferation is the presence of a smarter-than-human advisor who can say “no really human, if you fail to follow through, these will be the very likely results, and you won’t like them”.
I also hope that smarter-than-human advisor will say something like “look guys, you can all get vastly wealthier and longer-lived if you can just not freak out and fight each other”—and be so obviously right and convincing that humans will actually listen. The win-win solutions may just be compelling. I fully agree that no amount of sharing will prevent others from pursuing AGI—but generous sharing of technological benefits would reduce the priority of those efforts and the animosity when they’re thwarted.
Now is the time to think this through carefully, before the US commits to a race.
One issue with “explicitly and repeatedly saying what they will do” is that it invites competition. Many of the things that China hawks might want to do would be outside the Overton window. As Eliezer describes in AGI ruin: