Thinking out loud about some arguments about AI takeoff continuity:
If a discontinuous takeoff is more likely to be local to a particular agent or closely related set of agents with particular goals, and a continuous takeoff is more likely to be global, that seems like it incentivizes the first agent capable of creating a takeoff to make sure that that takeoff is discontinuous, so that it can reap the benefits of the takeoff being local to that agent. This seems like an argument for expecting a discontinuous takeoff and an important difference with other allegedly analogous technologies.
I have some trouble understanding the “before there are strongly self-improving AIs there will be moderately self-improving AIs” argument for continuity. Is there any reason to think the moderate self-improvement ability won’t be exactly what leads to the strong self-improvement ability? Before there’s an avalanche, there’s probably a smaller avalanche, but maybe the small avalanche is simply identical to the early part of the large avalanche.
Thinking out loud about some arguments about AI takeoff continuity:
If a discontinuous takeoff is more likely to be local to a particular agent or closely related set of agents with particular goals, and a continuous takeoff is more likely to be global, that seems like it incentivizes the first agent capable of creating a takeoff to make sure that that takeoff is discontinuous, so that it can reap the benefits of the takeoff being local to that agent. This seems like an argument for expecting a discontinuous takeoff and an important difference with other allegedly analogous technologies.
I have some trouble understanding the “before there are strongly self-improving AIs there will be moderately self-improving AIs” argument for continuity. Is there any reason to think the moderate self-improvement ability won’t be exactly what leads to the strong self-improvement ability? Before there’s an avalanche, there’s probably a smaller avalanche, but maybe the small avalanche is simply identical to the early part of the large avalanche.
Where have these points been discussed in depth?