Yes; except that a successful act can still be quite difficult.
You could reframe the concern to be that pivotal acts in a slow takeoff are prone to be bloody and dangerous. And because they are, and humans are likely to retain control, a pivotal act may be put off until it’s even more bloody—like a nuclear conflict or sending the sun nova.
Worse yet, the “pivotal act” may be performed by the worst (human) actor, not the best.
You are right that the same capabilities enable a pivotal act. My concern is that they won’t be used for one (where pivotal act is defined as a good act).
Having thought about it some more, I think the biggest problem in the multipolar, human-controlled RSI-capable AGI scenario is that it tends to be the worst actor that defects first and controls the future.
More ethical humans will tend to be more timid with committing or risking mass destruction to achieve their ends, so they’ll tend to hold off on aggressive moves that could win.
“Hide and create a superbrain and a robot army” are not the first things a good person tells their AGI to do, let alone inducing nuclear strikes that increase one’s odds of winning at great cost. Someone with more selfish designs on the future may have much less trouble issuing those orders.
Yes; except that a successful act can still be quite difficult.
You could reframe the concern to be that pivotal acts in a slow takeoff are prone to be bloody and dangerous. And because they are, and humans are likely to retain control, a pivotal act may be put off until it’s even more bloody—like a nuclear conflict or sending the sun nova.
Worse yet, the “pivotal act” may be performed by the worst (human) actor, not the best.
Just to elaborate a little:
You are right that the same capabilities enable a pivotal act. My concern is that they won’t be used for one (where pivotal act is defined as a good act).
Having thought about it some more, I think the biggest problem in the multipolar, human-controlled RSI-capable AGI scenario is that it tends to be the worst actor that defects first and controls the future.
More ethical humans will tend to be more timid with committing or risking mass destruction to achieve their ends, so they’ll tend to hold off on aggressive moves that could win.
“Hide and create a superbrain and a robot army” are not the first things a good person tells their AGI to do, let alone inducing nuclear strikes that increase one’s odds of winning at great cost. Someone with more selfish designs on the future may have much less trouble issuing those orders.