I think the idea is that once influence-seeking systems gain a certain amount of influence, it may become faster or more certain for them to gain more influence by causing a catastrophe than to continue to work within existing rules and institutions.
The key issue here is whether there will be coordination between a set of influence-seeking systems that can cause (and will benefit from) a catastrophe, even when other systems are opposing them. If we picture systems as having power comparable to what companies have now, that seems difficult. If we picture them as having power comparable to what countries have now, that seems fairly easy.
The key issue here is whether there will be coordination between a set of influence-seeking systems that can cause (and will benefit from) a catastrophe, even when other systems are opposing them.
Do you not expect this threshold to be crossed sooner or later, assuming AI alignment remains unsolved? Also, it seems like the main alternative to this scenario is that the influence-seeking systems expect to eventually gain control of most of the universe anyway (even without a “correlated automation failure”), so they don’t see a reason to “rock the boat” and try to dispossess humans of their remaining influence/power/resources, but this is almost as bad as the “correlated automation failure” scenario from an astronomical waste perspective. (I’m wondering if you’re questioning whether things will turn out badly, or questioning whether things will turn out badly this way.)
Mostly I am questioning whether things will turn out badly this way.
Do you not expect this threshold to be crossed sooner or later, assuming AI alignment remains unsolved?
Probably, but I’m pretty uncertain about this. It depends on a lot of messy details about reality, things like: how offense-defence balance scales; what proportion of powerful systems are mostly aligned; whether influence-seeking systems are risk-neutral; what self-governance structures they’ll set up; the extent to which their preferences are compatible with ours; how human-comprehensible the most important upcoming scientific advances are.
The key issue here is whether there will be coordination between a set of influence-seeking systems that can cause (and will benefit from) a catastrophe, even when other systems are opposing them. If we picture systems as having power comparable to what companies have now, that seems difficult. If we picture them as having power comparable to what countries have now, that seems fairly easy.
Do you not expect this threshold to be crossed sooner or later, assuming AI alignment remains unsolved? Also, it seems like the main alternative to this scenario is that the influence-seeking systems expect to eventually gain control of most of the universe anyway (even without a “correlated automation failure”), so they don’t see a reason to “rock the boat” and try to dispossess humans of their remaining influence/power/resources, but this is almost as bad as the “correlated automation failure” scenario from an astronomical waste perspective. (I’m wondering if you’re questioning whether things will turn out badly, or questioning whether things will turn out badly this way.)
Mostly I am questioning whether things will turn out badly this way.
Probably, but I’m pretty uncertain about this. It depends on a lot of messy details about reality, things like: how offense-defence balance scales; what proportion of powerful systems are mostly aligned; whether influence-seeking systems are risk-neutral; what self-governance structures they’ll set up; the extent to which their preferences are compatible with ours; how human-comprehensible the most important upcoming scientific advances are.