Realistically I think the core issue is that Eliezer is very skeptical about the possibility of competitive AI alignment. That said, I think that even on Eliezer’s pessimistic view he should probably just be complaining about competitiveness problems rather than saying pretty speculative stuff about what is needed for a pivotal act.
Isn’t the core thing here that Eliezer expects that a local, hard-takeoff is possible? He thinks that a single AI system can rapidly gain enormous power relative to the rest of the world (either by recursive self improvement, or by seizing compute, or by just deploying on more computers)
If this is possible thing for an AGI system to do, it seems like ensuring a human future requires that you’re able to prevent an unaligned AGI from undergoing a hard takeoff.
If you have aligned systems that are competitive in a number of different domains, that doesn’t matter if 1) local hard takeoff is on the table and 2) you aren’t able to produce systems whose alignment is robust to a hard takeoff.
It seems like the pivotal act ideology is a natural consequence of 1) expecting hard takeoff and 2) thinking that alignment is hard, full stop. Whether or not aligned systems will be competitive doesn’t come into it. Or by “competitive” do you mean, specifically “competitive, even across the huge relative capability gain of a hard takeoff”?
It seems like Eliezer’s chain of argument is:
[Hard takeoff is likely]
=>
[You need a pivotal act to preempt unaligned superintelligence]
=>
[Your safe AI design needs to be able to do something concrete that can enable a pivotal act in order to be of strategic relevance.]
=>
[When doing AI safety work, you need to be thinking about the concrete actions that your system will do]
Isn’t the core thing here that Eliezer expects that a local, hard-takeoff is possible? He thinks that a single AI system can rapidly gain enormous power relative to the rest of the world (either by recursive self improvement, or by seizing compute, or by just deploying on more computers)
If this is possible thing for an AGI system to do, it seems like ensuring a human future requires that you’re able to prevent an unaligned AGI from undergoing a hard takeoff.
If you have aligned systems that are competitive in a number of different domains, that doesn’t matter if 1) local hard takeoff is on the table and 2) you aren’t able to produce systems whose alignment is robust to a hard takeoff.
It seems like the pivotal act ideology is a natural consequence of 1) expecting hard takeoff and 2) thinking that alignment is hard, full stop. Whether or not aligned systems will be competitive doesn’t come into it. Or by “competitive” do you mean, specifically “competitive, even across the huge relative capability gain of a hard takeoff”?
It seems like Eliezer’s chain of argument is:
[Hard takeoff is likely]
=>
[You need a pivotal act to preempt unaligned superintelligence]
=>
[Your safe AI design needs to be able to do something concrete that can enable a pivotal act in order to be of strategic relevance.]
=>
[When doing AI safety work, you need to be thinking about the concrete actions that your system will do]