Condition 2 is necessary for race dynamics to arise, which is what people are usually worried about.
Suppose that AI systems weren’t going to be useful for anything—the only effect of AI systems was that they posed an x-risk to the world. Then it would still be true that “neither side wants to do the thing, because if they do the thing they get destroyed too”.
Nonetheless, I think in this world, no one ever builds AI systems and so don’t need to worry about x-risk.
That seems reasonable to me. I think what I’m thinking is that that’s a disanalogy between a potential “race” for transformative AI, and the race/motivation for building the first nuclear weapons, rather than a disanalogy between the AI situation and MAD.
So it seems like this disanalogy is a reason to think that the evidene “we built nuclear weapons” is weaker evidence than one might otherwise think for the claim “we’ll build dangerous AI” or the claim “we’ll build AI so in an especially ‘racing’/risky way”. And that seems an important point.
But it seems like “MAD strategies have been used” remains however strong evidence it previously was for the claim “we’ll do dangerous things with AI”. E.g., MAD strategies could still serve as some evidence for the general idea that countries/institutions are sometimes willing to do things that are risky to themselves, and that pose very large negative externalities of risks to others, for strategic reasons. And that general idea still seems to apply at least somewhat to AI.
(I’m not sure this is actually disagreeing with what you meant/believe.)
Suppose you have two events X and Y, such that X causes Y, that is, if not-X were true than not-Y would also be true.
Now suppose there’s some Y’ analogous to Y, and you make the argument A: “since Y happened, Y’ is also likely to happen”. If that’s all you know, I agree that A is reasonable evidence that Y’ is likely to happen. But if you then show that the analogous X’ is not true, while X was true, I think argument A provides ~no evidence.
Example:
“It was raining yesterday, so it will probably rain today.”
“But it was cloudy yesterday, and today it is sunny.”
“Ah. In that case it probably won’t rain.”
I think condition 2 causes racing causes MAD strategies in the case of nuclear weapons; since condition 2 / racing doesn’t hold in the case of AI, the fact that MAD strategies were used for nuclear weapons provides very little evidence about whether similar strategies will be used for AI.
MAD strategies could still serve as some evidence for the general idea that countries/institutions are sometimes willing to do things that are risky to themselves, and that pose very large negative externalities of risks to others, for strategic reasons.
I agree with that sentence interpreted literally. But I think you can change “for strategic reasons” to “in cases where condition 2 holds” and still capture most of the cases in which this happens.
I think I get what you’re saying. Is it roughly the following?
“If an AI race did occur, maybe similar issues to what we saw in MAD might occur; there may well be an analogy there. But there’s a disanalogy between the nuclear weapon case and the AI risk case with regards to the initial race, such that the initial nuclear race provides little/no evidence that a similar AI race may occur. And if a similar AI race doesn’t occur, then the conditions under which MAD-style strategies may arise would not occur. So it might not really matter if there’s an analogy between the AI risk situation if a race occurred and the MAD situation.”
If so, I think that makes sense to me, and it seems an interesting/important argument. Though it seems to suggest something more like “We may be more ok than people might think, as long as we avoid an AI race, and we’ll probably avoid an AI race”, rather than simply “We may be more ok than people might think”. And that distinction might e.g. suggest additional value to strategy/policy/governance work to avoid race dynamics, or to investigate how likely they are. (I don’t think this is disagreeing with you, just highlighting a particular thing a bit more.)
Condition 2 is necessary for race dynamics to arise, which is what people are usually worried about.
Suppose that AI systems weren’t going to be useful for anything—the only effect of AI systems was that they posed an x-risk to the world. Then it would still be true that “neither side wants to do the thing, because if they do the thing they get destroyed too”.
Nonetheless, I think in this world, no one ever builds AI systems and so don’t need to worry about x-risk.
That seems reasonable to me. I think what I’m thinking is that that’s a disanalogy between a potential “race” for transformative AI, and the race/motivation for building the first nuclear weapons, rather than a disanalogy between the AI situation and MAD.
So it seems like this disanalogy is a reason to think that the evidene “we built nuclear weapons” is weaker evidence than one might otherwise think for the claim “we’ll build dangerous AI” or the claim “we’ll build AI so in an especially ‘racing’/risky way”. And that seems an important point.
But it seems like “MAD strategies have been used” remains however strong evidence it previously was for the claim “we’ll do dangerous things with AI”. E.g., MAD strategies could still serve as some evidence for the general idea that countries/institutions are sometimes willing to do things that are risky to themselves, and that pose very large negative externalities of risks to others, for strategic reasons. And that general idea still seems to apply at least somewhat to AI.
(I’m not sure this is actually disagreeing with what you meant/believe.)
Suppose you have two events X and Y, such that X causes Y, that is, if not-X were true than not-Y would also be true.
Now suppose there’s some Y’ analogous to Y, and you make the argument A: “since Y happened, Y’ is also likely to happen”. If that’s all you know, I agree that A is reasonable evidence that Y’ is likely to happen. But if you then show that the analogous X’ is not true, while X was true, I think argument A provides ~no evidence.
Example:
“It was raining yesterday, so it will probably rain today.”
“But it was cloudy yesterday, and today it is sunny.”
“Ah. In that case it probably won’t rain.”
I think condition 2 causes racing causes MAD strategies in the case of nuclear weapons; since condition 2 / racing doesn’t hold in the case of AI, the fact that MAD strategies were used for nuclear weapons provides very little evidence about whether similar strategies will be used for AI.
I agree with that sentence interpreted literally. But I think you can change “for strategic reasons” to “in cases where condition 2 holds” and still capture most of the cases in which this happens.
I think I get what you’re saying. Is it roughly the following?
“If an AI race did occur, maybe similar issues to what we saw in MAD might occur; there may well be an analogy there. But there’s a disanalogy between the nuclear weapon case and the AI risk case with regards to the initial race, such that the initial nuclear race provides little/no evidence that a similar AI race may occur. And if a similar AI race doesn’t occur, then the conditions under which MAD-style strategies may arise would not occur. So it might not really matter if there’s an analogy between the AI risk situation if a race occurred and the MAD situation.”
If so, I think that makes sense to me, and it seems an interesting/important argument. Though it seems to suggest something more like “We may be more ok than people might think, as long as we avoid an AI race, and we’ll probably avoid an AI race”, rather than simply “We may be more ok than people might think”. And that distinction might e.g. suggest additional value to strategy/policy/governance work to avoid race dynamics, or to investigate how likely they are. (I don’t think this is disagreeing with you, just highlighting a particular thing a bit more.)
Yup, I agree with that summary.