What you wrote is not misleading at all. For example, I was able to glean that you are thinking over a timeline of a century, and generally agree with MIRI’s model of the AI problem. My objection to p(doom) is that none of those crucial details were included in the numbers themselves. In practice, the only interpretation I am seeing people actually use it for is as a signpost for whether someone is doomer or e/acc.
More specifically, p(doom) fails to communicate anything useful because there is so much uncertainty in the models. The differences in the probabilities people assign has a lot less to do with different assessments of the same thing than they do with assessing wildly different things.
Consider for a moment an alternative system, where we don’t talk about probability at all. Instead, we can take a couple of important dimensions of the alignment problem: I propose that these two dimensions be timelines, which is to say whether you expect a dangerous AGI to arrive in a short time or in a long time, and tractability of alignment, which is whether you expect aligning an AGI to be hard or to be easy. If all we asked people was which side of the origin they were on in both dimensions, we could symbolize this as + (meaning we have a lot of time, or alignment is an easy problem) and - (meaning we don’t have a lot of time, or alignment is a hard problem). This gives us four available answers:
doom++
doom+-
doom-+
doom--
I claim this gives us much more information than p(doom)=.97 because a scalar number tells you nothing about why. The positive and negative on dimensions give us a two-dimensional why: doom—means the alignment problem is hard and we have very little time in which to solve it. It neatly breaks down into a four quadrant graph for visually representing fundamental areas of agreement/disagreement.
In short, the probability calculations have no useful meaning unless they are being run on at least similar models; what we should be doing instead is finding ways to expose our personal models clearly and quickly. I think this will lead to better conversations, even in such limited domains as twitter.
What you wrote is not misleading at all. For example, I was able to glean that you are thinking over a timeline of a century, and generally agree with MIRI’s model of the AI problem. My objection to p(doom) is that none of those crucial details were included in the numbers themselves. In practice, the only interpretation I am seeing people actually use it for is as a signpost for whether someone is doomer or e/acc.
More specifically, p(doom) fails to communicate anything useful because there is so much uncertainty in the models. The differences in the probabilities people assign has a lot less to do with different assessments of the same thing than they do with assessing wildly different things.
Consider for a moment an alternative system, where we don’t talk about probability at all. Instead, we can take a couple of important dimensions of the alignment problem: I propose that these two dimensions be timelines, which is to say whether you expect a dangerous AGI to arrive in a short time or in a long time, and tractability of alignment, which is whether you expect aligning an AGI to be hard or to be easy. If all we asked people was which side of the origin they were on in both dimensions, we could symbolize this as + (meaning we have a lot of time, or alignment is an easy problem) and - (meaning we don’t have a lot of time, or alignment is a hard problem). This gives us four available answers:
doom++
doom+-
doom-+
doom--
I claim this gives us much more information than p(doom)=.97 because a scalar number tells you nothing about why. The positive and negative on dimensions give us a two-dimensional why: doom—means the alignment problem is hard and we have very little time in which to solve it. It neatly breaks down into a four quadrant graph for visually representing fundamental areas of agreement/disagreement.
In short, the probability calculations have no useful meaning unless they are being run on at least similar models; what we should be doing instead is finding ways to expose our personal models clearly and quickly. I think this will lead to better conversations, even in such limited domains as twitter.