I want to stress how I hugely like this post. What to do once we have an aligned AI of takeover level, or how to make sure no one will build an unaligned AI of takeover level, is in my opinion the biggest gap in many AI plans. I think answering this question might point to filling gaps that are currently completely unactioned, and I therefore really like this discussion. I previously tried to contribute to arguably the same question in this post, where I’m arguing that a pivotal act seems unlikely and therefore conclude that policy rather than alignment is likely to make sure we don’t go extinct.
They’d use their AGI to enforce that moratorium, along with hopefully minimal force.
I would say this is a pivotal act, although I like the sound of enforcing a moratorium better (and the opening it perhaps gives to enforcing a moratorium in the traditional, imo much preferred way of international policy).
I’m hereby providing a few reasons why I think a pivotal act might not happen:
A pivotal act is illegal. One needs to break into other people’s and other countries’ computer systems and do physical harm to property or possibly even people to enact it. Companies such as OpenAI and Anthropic are, although I’m not always a fan of them, generally law-abiding. It will be a big step for their leadership to do something as blatantly unlawful as a pivotal act.
There is zero indication that labs are planning to do a pivotal act. This may obviously have something to do with the point above, however, one would have expected hints from someone like Sam Altman who is hinting all the time, or leaks from people lower in the labs, if they were planning to do this.
The pivotal act is currently not even discussed seriously among experts and in fact highly unpopular in the discourse (see for example here).
If the labs are currently not planning to do this, it seems quite likely they won’t when the time comes.
Governments, especially the US government/ military, seem more likely in my opinion to perform a pivotal act. I’m not sure they will call it a pivotal act or necessarily have an existential reason in mind while performing it. They might see this as blocking adversaries from being able to attack the US, very much in their Overton window. However, for them as well, there is no certainty they would actually do this. There are large downsides: it is a hostile act towards another country, it could trigger conflict, they are likely to be uncertain how necessary this is at all, and uncertain what the progress is of an adversary project (perhaps underestimating it). For perhaps similar reasons, the US has not blocked the USSR atomic project before they had the bomb, even though this could have arguably preserved a unipolar instead of multipolar world order. Additionally, it is far from certain the US government will nationalize labs before they reach takeover level. Currently, there is little indication they will. I think it’s unreasonable to place more than say 80% confidence in the US government or military successfully blocking all adversaries’ projects before they reach takeover level.
I think it’s not unlikely that once an AI is powerful enough for a pivotal act, it will also be powerful enough to generally enforce hegemony, and not unlikely this will be persistent. I would be strongly against one country, or even lab, proclaiming and enforcing global hegemony for eternity. The risk that this might happen is a valid reason to support a pause, imo. If we get that lucky, I would much prefer a positive offense defense balance and many actors having AGI, while maintaining a power balance.
I think it’s too early to contribute to aligned ASI projects (Manhattan/CERN/Apollo/MAGIC/commercial/govt projects) as long as these questions are not resolved. For the moment, pushing for e.g. a conditional AI safety treaty is much more prudent, imo.
I want to stress how I hugely like this post. What to do once we have an aligned AI of takeover level, or how to make sure no one will build an unaligned AI of takeover level, is in my opinion the biggest gap in many AI plans. I think answering this question might point to filling gaps that are currently completely unactioned, and I therefore really like this discussion. I previously tried to contribute to arguably the same question in this post, where I’m arguing that a pivotal act seems unlikely and therefore conclude that policy rather than alignment is likely to make sure we don’t go extinct.
I would say this is a pivotal act, although I like the sound of enforcing a moratorium better (and the opening it perhaps gives to enforcing a moratorium in the traditional, imo much preferred way of international policy).
I’m hereby providing a few reasons why I think a pivotal act might not happen:
A pivotal act is illegal. One needs to break into other people’s and other countries’ computer systems and do physical harm to property or possibly even people to enact it. Companies such as OpenAI and Anthropic are, although I’m not always a fan of them, generally law-abiding. It will be a big step for their leadership to do something as blatantly unlawful as a pivotal act.
There is zero indication that labs are planning to do a pivotal act. This may obviously have something to do with the point above, however, one would have expected hints from someone like Sam Altman who is hinting all the time, or leaks from people lower in the labs, if they were planning to do this.
The pivotal act is currently not even discussed seriously among experts and in fact highly unpopular in the discourse (see for example here).
If the labs are currently not planning to do this, it seems quite likely they won’t when the time comes.
Governments, especially the US government/ military, seem more likely in my opinion to perform a pivotal act. I’m not sure they will call it a pivotal act or necessarily have an existential reason in mind while performing it. They might see this as blocking adversaries from being able to attack the US, very much in their Overton window. However, for them as well, there is no certainty they would actually do this. There are large downsides: it is a hostile act towards another country, it could trigger conflict, they are likely to be uncertain how necessary this is at all, and uncertain what the progress is of an adversary project (perhaps underestimating it). For perhaps similar reasons, the US has not blocked the USSR atomic project before they had the bomb, even though this could have arguably preserved a unipolar instead of multipolar world order. Additionally, it is far from certain the US government will nationalize labs before they reach takeover level. Currently, there is little indication they will. I think it’s unreasonable to place more than say 80% confidence in the US government or military successfully blocking all adversaries’ projects before they reach takeover level.
I think it’s not unlikely that once an AI is powerful enough for a pivotal act, it will also be powerful enough to generally enforce hegemony, and not unlikely this will be persistent. I would be strongly against one country, or even lab, proclaiming and enforcing global hegemony for eternity. The risk that this might happen is a valid reason to support a pause, imo. If we get that lucky, I would much prefer a positive offense defense balance and many actors having AGI, while maintaining a power balance.
I think it’s too early to contribute to aligned ASI projects (Manhattan/CERN/Apollo/MAGIC/commercial/govt projects) as long as these questions are not resolved. For the moment, pushing for e.g. a conditional AI safety treaty is much more prudent, imo.