Fantastic post! I agree with most of it, but I notice that Eliezer’s post has a strong tone of “this is really actually important, the modal scenario is that we literally all die, people aren’t taking this seriously and I need more help”. More measured or academic writing, even when it agrees in principle, doesn’t have the same tone or feeling of urgency. This has good effects (shaking people awake) and bad effects (panic/despair), but it’s a critical difference and my guess is the effects are net positive right now.
I definitely agree that Eliezer’s list of lethalities hits many rhetorical and pedagogical beats that other people are not hitting and I’m definitely not hitting. I also agree that it’s worth having a sense of urgency given that there’s a good chance of all of us dying (though quantitatively my risk of losing control of the universe though this channel is more like 20% than 99.99%, and I think extinction is a bit less less likely still).
I’m not totally sure about the net effects of the more extreme tone, I empathize with both the case in favor and the case against. Here I’m mostly just trying to contribute to the project of “get to the bottom of what’s likely to happen and what should be done.”
I did start the post with a list of 19 agreements with Eliezer, including many of the claims that are most relevant to the urgency, in part so that I wouldn’t be misconstrued as arguing that everything is fine.
I really appreciate your including a number here, that’s useful info. Would love to see more from everyone in the future—I know it takes more time/energy and operationalizations are hard, but I’d vastly prefer to see the easier versions over no versions or norms in favor of only writing up airtight probabilities.
(I also feel much better on an emotional level hearing 20% from you, I would’ve guessed anywhere between 30 and 90%. Others in the community may be similar: I’ve talked to multiple people who were pretty down after reading Eliezer’s last few posts.)
“[...] I give different numbers on different days. Sometimes that’s because I’ve considered new evidence, but normally it’s just because these numbers are just an imprecise quantification of my belief that changes from day to day.”
If the tweet is credible, I am curious if this difference in p(doom) is due to the day-to-day fluctuations of your belief, or have you considered new evidence and your initial belief that p(doom) < 20% is outdated?
I clarified my views here because people kept misunderstanding or misquoting them.
The grandparent describes my probability that humans irreversibly lose control of AI systems, which I’m still guessing at 10-20%. I should probably think harder about this at some point and revise it, I have no idea which direction it will move.
I think the tweet you linked is referring to the probability for “humanity irreversibly messes up our future within 10 years of building human-level AI.” (It’s presented as “probability of AI killing everyone” which is not really right.)
I generally don’t know what people mean when they say p(doom). I think they probably imagine that the vast majority of existential risk from AI comes from loss of control, and that catastrophic loss of control necessarily leads to extinction, both of which seem hard to defend.
The problem with Eliezer’s recent posts (IMO) is not in how pessimistic they are, but in how they are actively insulting to the reader. EY might not realize that his writing is insulting, but in that case he should have an editor who just elides those insulting points. (And also s/Eliezer/I/g please.)
My sense is not that Eliezer is insulting, so much as mainstream (and neurotypical) discourse treats far too much as insulting. Might be a distinction without a difference in practice, but “insulting” is relative to viewpoint and context, not an objective matter.
Fantastic post! I agree with most of it, but I notice that Eliezer’s post has a strong tone of “this is really actually important, the modal scenario is that we literally all die, people aren’t taking this seriously and I need more help”. More measured or academic writing, even when it agrees in principle, doesn’t have the same tone or feeling of urgency. This has good effects (shaking people awake) and bad effects (panic/despair), but it’s a critical difference and my guess is the effects are net positive right now.
I definitely agree that Eliezer’s list of lethalities hits many rhetorical and pedagogical beats that other people are not hitting and I’m definitely not hitting. I also agree that it’s worth having a sense of urgency given that there’s a good chance of all of us dying (though quantitatively my risk of losing control of the universe though this channel is more like 20% than 99.99%, and I think extinction is a bit less less likely still).
I’m not totally sure about the net effects of the more extreme tone, I empathize with both the case in favor and the case against. Here I’m mostly just trying to contribute to the project of “get to the bottom of what’s likely to happen and what should be done.”
I did start the post with a list of 19 agreements with Eliezer, including many of the claims that are most relevant to the urgency, in part so that I wouldn’t be misconstrued as arguing that everything is fine.
I really appreciate your including a number here, that’s useful info. Would love to see more from everyone in the future—I know it takes more time/energy and operationalizations are hard, but I’d vastly prefer to see the easier versions over no versions or norms in favor of only writing up airtight probabilities.
(I also feel much better on an emotional level hearing 20% from you, I would’ve guessed anywhere between 30 and 90%. Others in the community may be similar: I’ve talked to multiple people who were pretty down after reading Eliezer’s last few posts.)
This recent tweet claims that your current p(doom) is 50%.
In another post, you mentioned:
If the tweet is credible, I am curious if this difference in p(doom) is due to the day-to-day fluctuations of your belief, or have you considered new evidence and your initial belief that p(doom) < 20% is outdated?
I clarified my views here because people kept misunderstanding or misquoting them.
The grandparent describes my probability that humans irreversibly lose control of AI systems, which I’m still guessing at 10-20%. I should probably think harder about this at some point and revise it, I have no idea which direction it will move.
I think the tweet you linked is referring to the probability for “humanity irreversibly messes up our future within 10 years of building human-level AI.” (It’s presented as “probability of AI killing everyone” which is not really right.)
I generally don’t know what people mean when they say p(doom). I think they probably imagine that the vast majority of existential risk from AI comes from loss of control, and that catastrophic loss of control necessarily leads to extinction, both of which seem hard to defend.
The problem with Eliezer’s recent posts (IMO) is not in how pessimistic they are, but in how they are actively insulting to the reader. EY might not realize that his writing is insulting, but in that case he should have an editor who just elides those insulting points. (And also s/Eliezer/I/g please.)
My sense is not that Eliezer is insulting, so much as mainstream (and neurotypical) discourse treats far too much as insulting. Might be a distinction without a difference in practice, but “insulting” is relative to viewpoint and context, not an objective matter.