Still, I feel like this post doesn’t give a good argument for disjunctivity. To show that the arguments for a scenario with no outside view are likely, it takes more than just describing a model which is internally disjunctive. There needs to be some reason why we should strongly expect there to not be some external variables that could cause the model not to apply.
Some examples of these, in addition to the competence of humanity, are that deep learning could hit a wall for decades, Moore’s Law could come to a halt, some anti-tech regulation could cripple AI research, or alignment could turn out to be easy (which itself contains several disjunctive possibilities). I haven’t thought about these, and don’t claim that any of them are likely, but the possibility of these or other unknown factors invalidating the model prevents me from updating to a very high P(doom). Some of this comes from it just being a toy model, but adding more detail to the model isn’t enough to notably reduce the possibility of the model being wrong from unconsidered factors.
A statement I’m very confident in is that no perpetual motion machines will be developed in the next century. I could make some disjunctive list of potential failure modes a perpetual motion machine could encounter, and thus conclude that their development is unlikely, but this wouldn’t describe the actual reason a perpetual motion machine is unlikely. The actual reason is that I’m aware of certain laws of physics which prevent any perpetual motion machines from working, including ones with mechanisms wildly beyond my imagination. The outside view is another tool I can use to be very confident: I’m very confident that the next flight I take won’t crash, not because of my model of planes, but because any crash scenario which non-negligible probability would have caused some of the millions of commercial flights every year to crash, and that hasn’t happened. Avoiding AGI doom is not physically impossible and there is no outside view against it, and without some similarly compelling reason I can’t see how very high P(doom) can be justified.
Depends on what you mean by very high. If you mean >95% I agree with you. If you mean >50% I don’t.
Deep learning hits a wall for decades: <5% chance. I’m being generous here.
Moore’s law comes to a halt: Even if the price of compute stopped falling tomorrow, it would only push my timelines back a few years. (It would help a lot for >20 year timeline scenarios, but it wouldn’t be a silver bullet for them either.)
Anti-tech regulation being sufficiently strong, sufficiently targeted, and happening sufficiently soon that it actually prevents doom: This one I’m more optimistic about, but I still feel like it’s <10% chance by default.
Alignment turning out to be easy: I’m also somewhat hopeful about this one but still I give it <10% chance.
Analogy: Suppose it was 2015 and the question we were debating was “Will any humans be killed by poorly programmed self-driving cars?” A much lower-stakes question but analogous in a bunch of ways.
You could trot out a similar list of maybes to argue that the probability is <95%. Maybe deep learning will hit a wall and self-driving cars won’t be built, maybe making them recognize and avoid pedestrians will turn out to be easy, etc. But it would be wrong to conclude that the probability was therefore <50%.
I’m definitely only talking about probabilities in the range of >90%. >50% is justifiable without a strong argument for the disjunctivity of doom.
I like the self-driving car analogy, and I do think the probability in 2015 that a self-driving car would ever kill someone was between 50% and 95% (mostly because of a >5% chance that AGI comes before self-driving cars).
I’m really glad that this post is addressing the disjunctivity of AI doom, as my impression is that it is more of a crux than any of the reasons in https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities.
Still, I feel like this post doesn’t give a good argument for disjunctivity. To show that the arguments for a scenario with no outside view are likely, it takes more than just describing a model which is internally disjunctive. There needs to be some reason why we should strongly expect there to not be some external variables that could cause the model not to apply.
Some examples of these, in addition to the competence of humanity, are that deep learning could hit a wall for decades, Moore’s Law could come to a halt, some anti-tech regulation could cripple AI research, or alignment could turn out to be easy (which itself contains several disjunctive possibilities). I haven’t thought about these, and don’t claim that any of them are likely, but the possibility of these or other unknown factors invalidating the model prevents me from updating to a very high P(doom). Some of this comes from it just being a toy model, but adding more detail to the model isn’t enough to notably reduce the possibility of the model being wrong from unconsidered factors.
A statement I’m very confident in is that no perpetual motion machines will be developed in the next century. I could make some disjunctive list of potential failure modes a perpetual motion machine could encounter, and thus conclude that their development is unlikely, but this wouldn’t describe the actual reason a perpetual motion machine is unlikely. The actual reason is that I’m aware of certain laws of physics which prevent any perpetual motion machines from working, including ones with mechanisms wildly beyond my imagination. The outside view is another tool I can use to be very confident: I’m very confident that the next flight I take won’t crash, not because of my model of planes, but because any crash scenario which non-negligible probability would have caused some of the millions of commercial flights every year to crash, and that hasn’t happened. Avoiding AGI doom is not physically impossible and there is no outside view against it, and without some similarly compelling reason I can’t see how very high P(doom) can be justified.
Depends on what you mean by very high. If you mean >95% I agree with you. If you mean >50% I don’t.
Deep learning hits a wall for decades: <5% chance. I’m being generous here. Moore’s law comes to a halt: Even if the price of compute stopped falling tomorrow, it would only push my timelines back a few years. (It would help a lot for >20 year timeline scenarios, but it wouldn’t be a silver bullet for them either.) Anti-tech regulation being sufficiently strong, sufficiently targeted, and happening sufficiently soon that it actually prevents doom: This one I’m more optimistic about, but I still feel like it’s <10% chance by default. Alignment turning out to be easy: I’m also somewhat hopeful about this one but still I give it <10% chance.
Analogy: Suppose it was 2015 and the question we were debating was “Will any humans be killed by poorly programmed self-driving cars?” A much lower-stakes question but analogous in a bunch of ways.
You could trot out a similar list of maybes to argue that the probability is <95%. Maybe deep learning will hit a wall and self-driving cars won’t be built, maybe making them recognize and avoid pedestrians will turn out to be easy, etc. But it would be wrong to conclude that the probability was therefore <50%.
I’m definitely only talking about probabilities in the range of >90%. >50% is justifiable without a strong argument for the disjunctivity of doom.
I like the self-driving car analogy, and I do think the probability in 2015 that a self-driving car would ever kill someone was between 50% and 95% (mostly because of a >5% chance that AGI comes before self-driving cars).