First, I would like to second that the world is incredibly small. It bears repeating. I am repeating it to myself to get courage to write this comment. Maybe this is obvious, but maybe it is not. It could be helpful.
Random thoughts on alleged OpenAI memo on selling AGI to highest bidder including China and Russia. This sounds plausible to me, because as I understand before the split with Anthropic OpenAI was very much “team humanity, not team USG or team CCP”. I think this should be understood in context that getting aligned AI is higher priority than geopolitical competition.
Random thoughts on AI labs and coup. Could Los Alamos coup? I mean, obviously no in the real timeline, they didn’t have delivery, none of bomber, ICBM, and nuclear submarine. Let’s just assume after the Trinity test they could unilaterally decide to put a newly produced nuke not yet delivered to the army on ICBM and point that to Washington DC. Can Los Alamos force Truman, say, to share the nuke with Soviet Union (which many scientists actually wanted)?
By assumption, Truman should surrender (even unconditionally), but it is hard to imagine he would. Nuclear threats not only need to be executable, it also needs to be understandable. Also Los Alamos would depend on enriched uranium supply chain which is large industry not under its control, physical security of Los Alamos is under army control and what if security guards just go into Technical area?
Applying this to OpenAI or possible OpenAI-in-the-desert, OpenAI would depend on trillion dollars cluster and its supply chain, large industry not under its control, and same physical security problem. How does OpenAI defend against tanks on the street of San Francisco? With ASI-controlled drones? Why does OpenAI conveniently happen to have drones and drone factories on premise?
I am trying to push back against “if you have ASI you are the government”. If the government is monopoly on violence, millions of perfectly coordinated Von Neumanns do not immediately overthrow USG, key word being immediately. Considering Von Neumann’s talk of nuking Moscow today instead of tomorrow and lunch instead of dinner it will be pretty quick, but it still takes time to have fabs and power plants and data centers and drone factories etc. Even if you use nanotechnology to build them, it still takes time to research nanotechnology.
Maybe they develop mind control level convincing argument and send it to key people (president, congress, NORAD, etc) or hack their iPhones and recursively down to security guards of fabs/power plants/data centers/drone factories. That may be quick enough. The point is that it is not obvious.
Random thoughts on Chinese AI researchers and immigration. US’s track record here is extremely bad, even with cold war. Do you know how China got nukes and missiles? US deported Qian Xuesen, MIT graduate, who founded JPL. He had US military ranks in WW2. He interrogated Werner von Braun for USG! Then USG decided Qian is a communist, which was completely ridiculous. Then Qian went back and worked for communists whoops. Let me quote Atomic Heritage Foundation:
Deporting Qian was the stupidest thing this country ever did. He was no more a communist than I was, and we forced him to go.
US would be well advised to avoid repeating this total fiasco. But I am not optimistic.
Maybe they develop mind control level convincing argument and send it to key people (president, congress, NORAD, etc) or hack their iPhones and recursively down to security guards of fabs/power plants/data centers/drone factories. That may be quick enough. The point is that it is not obvious.
That’s the sort of thing that’d happen, yes. As with all AI takeover scenarios, it likely wouldn’t go down like this specifically, but you can be sure that the ASI would achieve the goal it wants to achieve/was told to achieve if aligned. (And see this post for my model of how this class of concrete scenarios would actually look like.)
Having nukes is not really a good analogy for having an aligned ASI at your disposal, as far as taking over the world is concerned. Unless your terminal value is human extinction, you can’t really nuke the world into the state of your personal utopia. You can’t even use nukes as leverage to threaten people into building your utopia, because:
Some people are good enough at decision theory to ignore threats.
Coercing people in this way might not actually be part of your utopia.
Your “power” is brittle. You only have the threat of nuclear armageddon to fall back on, and you can still be defeated by e. g. clever infiltration and sabotage, or by taking over your supply chains, etc. (If you have overwhelming, utterly loyal military power and security in full generality, that’s a very different setup.)
None of those constraints apply to having an ASI at your disposal. An ASI would let you implement your values upon the cosmos fully and faithfully, and it’d give you the roadmap to getting there from here.
This is also precisely why Leopold’s talk of “checks and balances” as the reason why governments could be trusted with AGI falls apart. “The government” isn’t some sort of holistic entity, it’s a collection of individuals with their own incentives, sometimes quite monstrous incentives. In the current regime, it’s indeed checked-and-balanced to be mostly sort-of (not really) aligned to the public good. But that property is absolutely not robust to you giving unchecked power to any given subsystem in it!
I’m really quite baffled that Leopold doesn’t get this, given his otherwise excellent analysis of the “authoritarianism risks” associated with aligned ASIs in the hands of private companies and the CCP. Glad to see @Zvi pointing that out.
To your question of what to do if you are outmatched and you only have an ASI at your disposal, I think the most logical thing to do is “do what the ASI tells you to”. The problem is that we have no way of predicting the outcomes if there is truly an ASI in the room. If it’s a superintelligence it is going to have better suggestions than anything you can come up with.
I would say that the majority of the many smart, competent, motivated people I worked closely with in the years I spent in the tech industry were not born in the US. Our immigration policies are hugely flawed. Gathering in the best people from around the world is good for both establishing power and establishing peace. We should do much more of it.
First, I would like to second that the world is incredibly small. It bears repeating. I am repeating it to myself to get courage to write this comment. Maybe this is obvious, but maybe it is not. It could be helpful.
Random thoughts on alleged OpenAI memo on selling AGI to highest bidder including China and Russia. This sounds plausible to me, because as I understand before the split with Anthropic OpenAI was very much “team humanity, not team USG or team CCP”. I think this should be understood in context that getting aligned AI is higher priority than geopolitical competition.
Random thoughts on AI labs and coup. Could Los Alamos coup? I mean, obviously no in the real timeline, they didn’t have delivery, none of bomber, ICBM, and nuclear submarine. Let’s just assume after the Trinity test they could unilaterally decide to put a newly produced nuke not yet delivered to the army on ICBM and point that to Washington DC. Can Los Alamos force Truman, say, to share the nuke with Soviet Union (which many scientists actually wanted)?
By assumption, Truman should surrender (even unconditionally), but it is hard to imagine he would. Nuclear threats not only need to be executable, it also needs to be understandable. Also Los Alamos would depend on enriched uranium supply chain which is large industry not under its control, physical security of Los Alamos is under army control and what if security guards just go into Technical area?
Applying this to OpenAI or possible OpenAI-in-the-desert, OpenAI would depend on trillion dollars cluster and its supply chain, large industry not under its control, and same physical security problem. How does OpenAI defend against tanks on the street of San Francisco? With ASI-controlled drones? Why does OpenAI conveniently happen to have drones and drone factories on premise?
I am trying to push back against “if you have ASI you are the government”. If the government is monopoly on violence, millions of perfectly coordinated Von Neumanns do not immediately overthrow USG, key word being immediately. Considering Von Neumann’s talk of nuking Moscow today instead of tomorrow and lunch instead of dinner it will be pretty quick, but it still takes time to have fabs and power plants and data centers and drone factories etc. Even if you use nanotechnology to build them, it still takes time to research nanotechnology.
Maybe they develop mind control level convincing argument and send it to key people (president, congress, NORAD, etc) or hack their iPhones and recursively down to security guards of fabs/power plants/data centers/drone factories. That may be quick enough. The point is that it is not obvious.
Random thoughts on Chinese AI researchers and immigration. US’s track record here is extremely bad, even with cold war. Do you know how China got nukes and missiles? US deported Qian Xuesen, MIT graduate, who founded JPL. He had US military ranks in WW2. He interrogated Werner von Braun for USG! Then USG decided Qian is a communist, which was completely ridiculous. Then Qian went back and worked for communists whoops. Let me quote Atomic Heritage Foundation:
US would be well advised to avoid repeating this total fiasco. But I am not optimistic.
That’s the sort of thing that’d happen, yes. As with all AI takeover scenarios, it likely wouldn’t go down like this specifically, but you can be sure that the ASI would achieve the goal it wants to achieve/was told to achieve if aligned. (And see this post for my model of how this class of concrete scenarios would actually look like.)
Having nukes is not really a good analogy for having an aligned ASI at your disposal, as far as taking over the world is concerned. Unless your terminal value is human extinction, you can’t really nuke the world into the state of your personal utopia. You can’t even use nukes as leverage to threaten people into building your utopia, because:
Some people are good enough at decision theory to ignore threats.
Coercing people in this way might not actually be part of your utopia.
Your “power” is brittle. You only have the threat of nuclear armageddon to fall back on, and you can still be defeated by e. g. clever infiltration and sabotage, or by taking over your supply chains, etc. (If you have overwhelming, utterly loyal military power and security in full generality, that’s a very different setup.)
None of those constraints apply to having an ASI at your disposal. An ASI would let you implement your values upon the cosmos fully and faithfully, and it’d give you the roadmap to getting there from here.
This is also precisely why Leopold’s talk of “checks and balances” as the reason why governments could be trusted with AGI falls apart. “The government” isn’t some sort of holistic entity, it’s a collection of individuals with their own incentives, sometimes quite monstrous incentives. In the current regime, it’s indeed checked-and-balanced to be mostly sort-of (not really) aligned to the public good. But that property is absolutely not robust to you giving unchecked power to any given subsystem in it!
I’m really quite baffled that Leopold doesn’t get this, given his otherwise excellent analysis of the “authoritarianism risks” associated with aligned ASIs in the hands of private companies and the CCP. Glad to see @Zvi pointing that out.
To your question of what to do if you are outmatched and you only have an ASI at your disposal, I think the most logical thing to do is “do what the ASI tells you to”. The problem is that we have no way of predicting the outcomes if there is truly an ASI in the room. If it’s a superintelligence it is going to have better suggestions than anything you can come up with.
I would say that the majority of the many smart, competent, motivated people I worked closely with in the years I spent in the tech industry were not born in the US. Our immigration policies are hugely flawed. Gathering in the best people from around the world is good for both establishing power and establishing peace. We should do much more of it.