unilateral pivotal acts, and government-backed governance efforts to slow down AI capabilities research and investment
There is nothing inherently unilateral about pivotal acts. The problem with an international moratorium is that with enforcement tools that are readily available it’s unlikely to last as long as it needs to for human-level alignment theory to catch up. Being government-backed is not part of the problem. Pivotal AI-enabled trajectories of development can help with that, by providing the tools for a more reliable international moratorium and for getting to a place where the field of alignment is actually ready for tackling more capability.
When I referred to pivotal acts, I implied the use of enforcement tools that are extremely powerful, of the sort implied in AGI Ruin. That is, enforcement tools that make an actual impact in extending timelines[1]. Perhaps I should start using a more precise term to describe this from now on.
It is hard for me to imagine how there can be consensus within a US government organization capable of launching a superhuman-enforcement-tool-based pivotal act (such as three letter agencies) to initiate a moratorium, much less consensus in the US government or between US and EU (especially given the rather interesting strategy EU is trying with their AI Act).
I continue to consider all superhuman-enforcement-tool-based pivotal acts as unilateral given this belief. My use of the world “unilateral” points to the fact that the organizations and people who currently have a non-trivial influence over the state of the world and its future will almost entirely be blindsided by the pivotal act, and that will result in destruction of trust and chaos and an increase in conflict. And I currently believe that this is actually more likely to increase P(doom) or existential risk for humanity, even if it extends the foom timeline.
Although not preventing ASI creation entirely. The destruction of humanity’s potential is also an existential risk, and the inability for us to create a utopia is too painful to bear.
There is nothing physically impossible about it lasting however long it needs to, that’s only implausible for the same political and epistemic reasons that any global moratorium at all is implausible. GPUs don’t grow on trees.
My point in the above comment is that pivotal acts don’t by their nature stay apart, a conventional moratorium that actually helps is also a pivotal act. Pivotal act AIs are something like task AIs that can plausibly be made to achieve a strategically relevant effect relatively safely, well in advance of actually having an understanding necessary to align a general agentic superintelligence, using alignment techniques designed around lack of such an understanding. Advances made by humans with use of task AIs could then increase robustness of a moratorium’s enforcement (better cybersecurity and compute governance), reduce the downsides of the moratorium’s presence (tool AIs allowed to make biotech advancements), and ultimately move towards being predictably ready for a superintelligent AI, which might initially look like developing alignment techniques that work for making more and more powerful task AIs safely. Scalable molecular manufacturing of compute is an obvious landmark, and can’t end well without robust compute governance already in place. Human uploading is another tool that can plausibly be used to improve global security without having a better understanding of AI alignment.
(I don’t see what we currently know justifying Hanson’s concern of never making enough progress to lift a value drift moratorium. If theoretical progress can get feedback from gradually improving task AIs, there is a long way to go before concluding that the process would peter out before superintelligence, so that taking any sort of plunge is remotely sane for the world. We haven’t been at it for even a million years yet.)
There is nothing inherently unilateral about pivotal acts. The problem with an international moratorium is that with enforcement tools that are readily available it’s unlikely to last as long as it needs to for human-level alignment theory to catch up. Being government-backed is not part of the problem. Pivotal AI-enabled trajectories of development can help with that, by providing the tools for a more reliable international moratorium and for getting to a place where the field of alignment is actually ready for tackling more capability.
When I referred to pivotal acts, I implied the use of enforcement tools that are extremely powerful, of the sort implied in AGI Ruin. That is, enforcement tools that make an actual impact in extending timelines[1]. Perhaps I should start using a more precise term to describe this from now on.
It is hard for me to imagine how there can be consensus within a US government organization capable of launching a superhuman-enforcement-tool-based pivotal act (such as three letter agencies) to initiate a moratorium, much less consensus in the US government or between US and EU (especially given the rather interesting strategy EU is trying with their AI Act).
I continue to consider all superhuman-enforcement-tool-based pivotal acts as unilateral given this belief. My use of the world “unilateral” points to the fact that the organizations and people who currently have a non-trivial influence over the state of the world and its future will almost entirely be blindsided by the pivotal act, and that will result in destruction of trust and chaos and an increase in conflict. And I currently believe that this is actually more likely to increase P(doom) or existential risk for humanity, even if it extends the foom timeline.
Although not preventing ASI creation entirely. The destruction of humanity’s potential is also an existential risk, and the inability for us to create a utopia is too painful to bear.
How long do you think such a moratorium would last?
There is nothing physically impossible about it lasting however long it needs to, that’s only implausible for the same political and epistemic reasons that any global moratorium at all is implausible. GPUs don’t grow on trees.
My point in the above comment is that pivotal acts don’t by their nature stay apart, a conventional moratorium that actually helps is also a pivotal act. Pivotal act AIs are something like task AIs that can plausibly be made to achieve a strategically relevant effect relatively safely, well in advance of actually having an understanding necessary to align a general agentic superintelligence, using alignment techniques designed around lack of such an understanding. Advances made by humans with use of task AIs could then increase robustness of a moratorium’s enforcement (better cybersecurity and compute governance), reduce the downsides of the moratorium’s presence (tool AIs allowed to make biotech advancements), and ultimately move towards being predictably ready for a superintelligent AI, which might initially look like developing alignment techniques that work for making more and more powerful task AIs safely. Scalable molecular manufacturing of compute is an obvious landmark, and can’t end well without robust compute governance already in place. Human uploading is another tool that can plausibly be used to improve global security without having a better understanding of AI alignment.
(I don’t see what we currently know justifying Hanson’s concern of never making enough progress to lift a value drift moratorium. If theoretical progress can get feedback from gradually improving task AIs, there is a long way to go before concluding that the process would peter out before superintelligence, so that taking any sort of plunge is remotely sane for the world. We haven’t been at it for even a million years yet.)