I expect that Peter and Jeremy aren’t particularly committed to covert and forceful takeover and they don’t think of this as a key conclusion.
Instead they care more about arguing about how resources will end up distributed in the long run.
If the claim is, for example, that AIs could own 99.99% of the universe, and humans will only own 0.01%, but all of us humans will be many orders of magnitude richer (because the universe is so big), and yet this still counts as a “catastrophe” because of the relative distribution of wealth and resources, I think that needs to be way more clear in the text.
I could imagine large collateral damage due to conflict over resources between AIs.
To be clear: I’m also very concerned about future AI conflict, and I think that if such a widespread conflict occurred (imagine: world war 3 but with robot armies in addition to nanotech and anti-matter bombs), I would be very worried, not only for my own life, but for the state of the world generally. My own view on this issue is simply that it is imprecise and approximately inaccurate to round such an problem off to generic problems of technical misalignment, relative to broader structural problems related to the breakdown of institutions designed to keep the peace among various parties in the world.
yet this is still counts as a “catastrophe” because of the relative distribution of wealth and resources, I think that needs to be way more clear in the text.
(But I think they do argue for violent conflict in text. It would probably be more clear if they were like “we mostly aren’t arguing for violent takeover or loss of human life here, though this has been discussed in more detail elsewhere”)
TBC, they discuss negative consequences of powerful, uncontrolled, and not-particularly-aligned AI in section 6, but they don’t argue for “this will result in violent conflict” in that much detail. I think the argument they make is basically right and suffices for thinking that the type of scenario they describe is reasonably likely to end in violent conflict (though more like 70% than 95% IMO). I just don’t see this as one of the main arguments of this post and probably isn’t a key crux for them.
I agree that it’d be extremely misleading if we defined “catastrophe” in a way that includes futures where everyone is better off than they currently are in every way (without being very clear about it). This is not what we mean by catastrophe.
If the claim is, for example, that AIs could own 99.99% of the universe, and humans will only own 0.01%, but all of us humans will be many orders of magnitude richer (because the universe is so big), and yet this still counts as a “catastrophe” because of the relative distribution of wealth and resources, I think that needs to be way more clear in the text.
To be clear: I’m also very concerned about future AI conflict, and I think that if such a widespread conflict occurred (imagine: world war 3 but with robot armies in addition to nanotech and anti-matter bombs), I would be very worried, not only for my own life, but for the state of the world generally. My own view on this issue is simply that it is imprecise and approximately inaccurate to round such an problem off to generic problems of technical misalignment, relative to broader structural problems related to the breakdown of institutions designed to keep the peace among various parties in the world.
Also, for the record, I totally agree with:
(But I think they do argue for violent conflict in text. It would probably be more clear if they were like “we mostly aren’t arguing for violent takeover or loss of human life here, though this has been discussed in more detail elsewhere”)
TBC, they discuss negative consequences of powerful, uncontrolled, and not-particularly-aligned AI in section 6, but they don’t argue for “this will result in violent conflict” in that much detail. I think the argument they make is basically right and suffices for thinking that the type of scenario they describe is reasonably likely to end in violent conflict (though more like 70% than 95% IMO). I just don’t see this as one of the main arguments of this post and probably isn’t a key crux for them.
I agree that it’d be extremely misleading if we defined “catastrophe” in a way that includes futures where everyone is better off than they currently are in every way (without being very clear about it). This is not what we mean by catastrophe.