@Max Tegmark my impression is that you believe that some amount of cooperation between the US and China is possible. If the US takes steps that show that it is willing to avoid an AGI race, then there’s some substantial probability that China will also want to avoid an AGI race. (And perhaps there could be some verification methods that support a “trust but verify” approach to international agreements.)
My main question: Are there circumstances under which you would no longer believe that cooperation is possible & you would instead find yourself advocating for an entente strategy?
When I look at the world right now, it seems to me like there’s so much uncertainty around how governments will react to AGI that I think it’s silly to throw out the idea of international coordination. As you mention in the post, there are also some signs that Chinese leaders and experts are concerned about AI risks.
It seems plausible to me that governments could– if they were sufficiently concerned about misalignment risks and believed in the assumptions behind calling an AGI race a “suicide race”– end up reaching cooperative agreements and pursuing some alternative to the “suicide race”
But suppose for sake of argument that there was compelling evidence that China was not willing to cooperate with the US. I don’t mean the kind of evidence we have now, and I think we both probably agree that many actors will have incentives to say “there’s no way China will cooperate with us” even in the absence of strong evidence. But if such evidence emerged, what do you think the best strategy would be from there? If hypothetically it became clear that China’s leadership were essentially taking an e/acc approach and were really truly interested in getting AGI ~as quickly as possible, what do you think should be done?
I ask partially because I’m trying to think more clearly about these topics myself. I think my current viewpoint is something like:
In general, the US should avoid taking actions that make a race with China more likely or inevitable.
The primary plan should be for the US to engage in good-faith efforts to pursue international coordination, aiming toward a world where there are verifiable ways to avoid the premature development of AGI.
We could end up in a scenario in which the prospect of international coordination has fallen apart. (e.g., China or some other major US adversary adopts a very “e/acc mindset” and seems to be gunning toward AGI with safety plans that are considerably worse than those proposed by the US.) At this point, it seems to me like the US would either have to (a) try to get to AGI before the adversary [essentially the Entente plan] or (b) give up and just kinda hope that the adversary ends up changing course as they get closer to AGI. Let’s call this “world #3″[1].
Again, I think a lot of folks will have strong incentives to try to paint us as being in world #3, and I personally don’t think we have enough evidence to say “yup, we’re so confident we’re in world #3 that we should go with an entente strategy.”. But I’m curious if you’ve thought about the conditions under which you’d conclude that we are quite confidently in world #3 and what you think we should do from there.
World #1: Status quo; governments are not sufficiently concerned; corporations race to develop AGI
World #2: Governments become quite concerned about AGI and pursue international coordination
World #3: Governments become quite concerned about AGI but there is strong evidence that at least one major world power is refusing to cooperate//gunning toward AGI.
Thanks Akash! As I mentioned in my reply to Nicholas, I view it as flawed to think that China or the US would only abstain from AGI because of a Sino-US agreement. Rather, they’d each unilaterally do it out of national self-interest.
It’s not in the US self-interest to disempower itself and all its current power centers by allowing a US company to build uncontrollable AGI.
It’s not in the interest of the Chinese Communist Party to disempower itself by allowing a Chinese company to build uncontrollable AGI.
Once the US and Chinese leadership serves their self-interest by preventing uncontrollable AGI at home, they have a shared incentive to coordinate to do the same globally. The reason that the self-interest hasn’t yet played out is that US and Chinese leaders still haven’t fully understood the game theory payout matrix: the well-funded and wishful-thinking-fueled disinformation campaign arguing that Turing, Hinton, Bengio, Russell, Yudkowski et al are wrong (that we’re likely to figure out to control AGI in time if we “scale quickly”) is massively successful. That success is unsurprising, given how successful the disinformation campaigns were for, e.g., tobacco, asbesthos and leaded gasoline – the only difference is that the stakes are much higher now.
@Max Tegmark my impression is that you believe that some amount of cooperation between the US and China is possible. If the US takes steps that show that it is willing to avoid an AGI race, then there’s some substantial probability that China will also want to avoid an AGI race. (And perhaps there could be some verification methods that support a “trust but verify” approach to international agreements.)
My main question: Are there circumstances under which you would no longer believe that cooperation is possible & you would instead find yourself advocating for an entente strategy?
When I look at the world right now, it seems to me like there’s so much uncertainty around how governments will react to AGI that I think it’s silly to throw out the idea of international coordination. As you mention in the post, there are also some signs that Chinese leaders and experts are concerned about AI risks.
It seems plausible to me that governments could– if they were sufficiently concerned about misalignment risks and believed in the assumptions behind calling an AGI race a “suicide race”– end up reaching cooperative agreements and pursuing some alternative to the “suicide race”
But suppose for sake of argument that there was compelling evidence that China was not willing to cooperate with the US. I don’t mean the kind of evidence we have now, and I think we both probably agree that many actors will have incentives to say “there’s no way China will cooperate with us” even in the absence of strong evidence. But if such evidence emerged, what do you think the best strategy would be from there? If hypothetically it became clear that China’s leadership were essentially taking an e/acc approach and were really truly interested in getting AGI ~as quickly as possible, what do you think should be done?
I ask partially because I’m trying to think more clearly about these topics myself. I think my current viewpoint is something like:
In general, the US should avoid taking actions that make a race with China more likely or inevitable.
The primary plan should be for the US to engage in good-faith efforts to pursue international coordination, aiming toward a world where there are verifiable ways to avoid the premature development of AGI.
We could end up in a scenario in which the prospect of international coordination has fallen apart. (e.g., China or some other major US adversary adopts a very “e/acc mindset” and seems to be gunning toward AGI with safety plans that are considerably worse than those proposed by the US.) At this point, it seems to me like the US would either have to (a) try to get to AGI before the adversary [essentially the Entente plan] or (b) give up and just kinda hope that the adversary ends up changing course as they get closer to AGI. Let’s call this “world #3″[1].
Again, I think a lot of folks will have strong incentives to try to paint us as being in world #3, and I personally don’t think we have enough evidence to say “yup, we’re so confident we’re in world #3 that we should go with an entente strategy.”. But I’m curious if you’ve thought about the conditions under which you’d conclude that we are quite confidently in world #3 and what you think we should do from there.
I sometimes think about the following situations:
World #1: Status quo; governments are not sufficiently concerned; corporations race to develop AGI
World #2: Governments become quite concerned about AGI and pursue international coordination
World #3: Governments become quite concerned about AGI but there is strong evidence that at least one major world power is refusing to cooperate//gunning toward AGI.
Thanks Akash! As I mentioned in my reply to Nicholas, I view it as flawed to think that China or the US would only abstain from AGI because of a Sino-US agreement. Rather, they’d each unilaterally do it out of national self-interest.
It’s not in the US self-interest to disempower itself and all its current power centers by allowing a US company to build uncontrollable AGI.
It’s not in the interest of the Chinese Communist Party to disempower itself by allowing a Chinese company to build uncontrollable AGI.
Once the US and Chinese leadership serves their self-interest by preventing uncontrollable AGI at home, they have a shared incentive to coordinate to do the same globally. The reason that the self-interest hasn’t yet played out is that US and Chinese leaders still haven’t fully understood the game theory payout matrix: the well-funded and wishful-thinking-fueled disinformation campaign arguing that Turing, Hinton, Bengio, Russell, Yudkowski et al are wrong (that we’re likely to figure out to control AGI in time if we “scale quickly”) is massively successful. That success is unsurprising, given how successful the disinformation campaigns were for, e.g., tobacco, asbesthos and leaded gasoline – the only difference is that the stakes are much higher now.