I largely agree with all these points, with my minor points of disagreement being insufficient to change the overall conclusions. I feel like an important point which should be emphasized more is that our best hope for saving humanity lies in maximizing the non-linearly-intelligence-weighted researcher hours invested in AGI safety research before the advent of the first dangerously powerful unaligned AGI. To maximize this key metric, we need to get more and smarter people doing this research, and we need to slow down AGI capabilities research. Insofar as AI Governance is a tactic worth pursuing, it must pursue one or both of these specific aims. Once dangerously powerful unaligned AGI has been launched, it’s too late for politics or social movements or anything slower than perhaps decisive military action prepped ahead of time (e.g. the secret AGI-prevention department hitting the detonation switch for all the secret prepared explosives in all the worlds’ data centers).
I largely agree with all these points, with my minor points of disagreement being insufficient to change the overall conclusions. I feel like an important point which should be emphasized more is that our best hope for saving humanity lies in maximizing the non-linearly-intelligence-weighted researcher hours invested in AGI safety research before the advent of the first dangerously powerful unaligned AGI. To maximize this key metric, we need to get more and smarter people doing this research, and we need to slow down AGI capabilities research. Insofar as AI Governance is a tactic worth pursuing, it must pursue one or both of these specific aims. Once dangerously powerful unaligned AGI has been launched, it’s too late for politics or social movements or anything slower than perhaps decisive military action prepped ahead of time (e.g. the secret AGI-prevention department hitting the detonation switch for all the secret prepared explosives in all the worlds’ data centers).