Thanks for the post! I see two important difficulties with your proposal.
First, you say (quoting your comment below)
It’s not in the US self-interest to disempower itself and all its current power centers by allowing a US company to build uncontrollable AGI...The reason that the self-interest hasn’t yet played out is that US and Chinese leaders still haven’t fully understood the game theory payout matrix.
The trouble here is that it is in the US (& China’s) self-interest, as that’s seen by some leaders, to take some chance of out-of-control AGI if the alternative is the other side taking over. And either country can create safety standards for consumer products while secretly pursuing AGI for military or other purposes. That changes the payout matrix dramatically.
I think your argument could work if
a) both sides could trust that the other was applying its safety standards universally, but that takes international cooperation rather than simple self-interest; or
b) it was common knowledge that AGI was highly likely to be uncontrollable, but now we’re back to the same debate about existential risk from AI that we were in before your proposal.
Second (and less centrally), as others have pointed out, your definition of tool AI as (in part) ‘AI that we can control’ begs the question. Certainly for some kinds of tool AI such as AlphaFold, it’s easy to show that we can control them; they only operate over a very narrow domain. But for broader sorts of tools like assistants to help us manage our daily tasks, which people clearly want and for which there are strong economic incentives, it’s not obvious what level of risk to expect, and again we’re back to the same debates we were already having.
A world with good safety standards for AI is certainly preferable to a world without them, and I think there’s value in advocating for them and in pointing out the risks in the just-scale-fast position. But I think this proposal fails to address some critical challenges of escaping the current domestic and international race dynamics.
Thanks for the post! I see two important difficulties with your proposal.
First, you say (quoting your comment below)
The trouble here is that it is in the US (& China’s) self-interest, as that’s seen by some leaders, to take some chance of out-of-control AGI if the alternative is the other side taking over. And either country can create safety standards for consumer products while secretly pursuing AGI for military or other purposes. That changes the payout matrix dramatically.
I think your argument could work if
a) both sides could trust that the other was applying its safety standards universally, but that takes international cooperation rather than simple self-interest; or
b) it was common knowledge that AGI was highly likely to be uncontrollable, but now we’re back to the same debate about existential risk from AI that we were in before your proposal.
Second (and less centrally), as others have pointed out, your definition of tool AI as (in part) ‘AI that we can control’ begs the question. Certainly for some kinds of tool AI such as AlphaFold, it’s easy to show that we can control them; they only operate over a very narrow domain. But for broader sorts of tools like assistants to help us manage our daily tasks, which people clearly want and for which there are strong economic incentives, it’s not obvious what level of risk to expect, and again we’re back to the same debates we were already having.
A world with good safety standards for AI is certainly preferable to a world without them, and I think there’s value in advocating for them and in pointing out the risks in the just-scale-fast position. But I think this proposal fails to address some critical challenges of escaping the current domestic and international race dynamics.