I start with a very low prior of AGI doom … I give 1 bit of evidence collectively for the weak arguments
Imagine that you have to argue with someone who believes in 50% doom[1] on priors. Then you’d need to articulate the reasons for adopting your priors. Priors are not arguments, it’s hard to put them into arguments. Sometimes priors are formed by exposure to arguments, but even then they are not necessarily captured by those arguments. Articulation of your priors might then look similarly tangential and ultimately unconvincing to those who don’t share them, if it doesn’t form a larger narrative that’s no longer a specific argument.
a very low pDoom (2-20%) compared to most people on LessWrong
Were there surveys that put most people on LW higher than that? My impression is that lower bounds of 10-20% are not too low for most, at least for doom in the sense of extinction.
Worth distinguishing doom in the sense of extinction and doom in the sense of existential risk short of extinction, getting most of the cosmic wealth taken away. I have very high doom expectations in the sense of loss of cosmic wealth, but only 20-40% for extinction.
Worth distinguishing doom in the sense of extinction and doom in the sense of existential risk short of extinction, getting most of the cosmic wealth taken away. I have very high doom expectations in the sense of loss of cosmic wealth, but only 20-40% for extinction.
By doom I mean the universe gets populated by AI with no moral worth (e.g. paperclippers). I expect humans to look pretty different in a century or two even if AGI was somehow impossible, so I don’t really care about preserving status-quo humanity.
I don’t really care about preserving status-quo humanity
By non-extinction I don’t mean freezing the status quo of necessarily biological Homo sapiens on Earth, though with ASI I expect that non-extinction in particular keeps most individual people alive indefinitely in the way they individually choose. I think this is a more natural reading of non-extinction (as a sense of non-doom) than a perpetual state of human nature preserve.
By doom I mean the universe gets populated by AI with no moral worth
So losing cosmic wealth is sufficient to qualify an outcome as doom, as in Bostrom’s existential risk. What if Earth literally remains untouched by machines, protected from extinction level events but otherwise left alone, is it still doom in this sense? What if aliens that hold moral worth but currently labor under an unkind regime exterminate humanity, but then the aliens themselves spread to the stars (taking them for themselves) and live happily ever after, is that still not a central example of doom?
My point is that the term is highly ambiguous, resolution criteria for predictions that involve it are all over the place, so it’s no good for use in predictions or communication. There is illusion of transparency where people keep expecting it to be understood, and then others incorrectly think that they’ve understood the intended meaning. Splitting doom into extinction and loss of cosmic wealth seems less ambiguous.
So losing cosmic wealth is sufficient to qualify an outcome as doom
My utility function roughly looks like:
my survival
the survival of the people I know and care about
the distant future is populated by beings that are in some way “descended” from humanity and share at least some of the values (love, joy, curiosity, creativity) that I currently hold
Basically, if I sat down with a human from 10,000 years ago, I think there’s a lot we would disagree about, but at the end of the day I think they would get the feeling that I’m an “okay person”. I would like to imagine the same sort of thing holding for whatever follows us.
I don’t find the hair-splitting arguments like “what if the AGI takes over the universe but leaves Earth intact” particularly interesting except insofar as it allows for all 3 of the above. I also don’t think most people have a huge faction of P(~doom) on such weird technicalities.
Imagine that you have to argue with someone who believes in 50% doom[1] on priors. Then you’d need to articulate the reasons for adopting your priors. Priors are not arguments, it’s hard to put them into arguments. Sometimes priors are formed by exposure to arguments, but even then they are not necessarily captured by those arguments. Articulation of your priors might then look similarly tangential and ultimately unconvincing to those who don’t share them, if it doesn’t form a larger narrative that’s no longer a specific argument.
Were there surveys that put most people on LW higher than that? My impression is that lower bounds of 10-20% are not too low for most, at least for doom in the sense of extinction.
Worth distinguishing doom in the sense of extinction and doom in the sense of existential risk short of extinction, getting most of the cosmic wealth taken away. I have very high doom expectations in the sense of loss of cosmic wealth, but only 20-40% for extinction.
By doom I mean the universe gets populated by AI with no moral worth (e.g. paperclippers). I expect humans to look pretty different in a century or two even if AGI was somehow impossible, so I don’t really care about preserving status-quo humanity.
By non-extinction I don’t mean freezing the status quo of necessarily biological Homo sapiens on Earth, though with ASI I expect that non-extinction in particular keeps most individual people alive indefinitely in the way they individually choose. I think this is a more natural reading of non-extinction (as a sense of non-doom) than a perpetual state of human nature preserve.
So losing cosmic wealth is sufficient to qualify an outcome as doom, as in Bostrom’s existential risk. What if Earth literally remains untouched by machines, protected from extinction level events but otherwise left alone, is it still doom in this sense? What if aliens that hold moral worth but currently labor under an unkind regime exterminate humanity, but then the aliens themselves spread to the stars (taking them for themselves) and live happily ever after, is that still not a central example of doom?
My point is that the term is highly ambiguous, resolution criteria for predictions that involve it are all over the place, so it’s no good for use in predictions or communication. There is illusion of transparency where people keep expecting it to be understood, and then others incorrectly think that they’ve understood the intended meaning. Splitting doom into extinction and loss of cosmic wealth seems less ambiguous.
My utility function roughly looks like:
my survival
the survival of the people I know and care about
the distant future is populated by beings that are in some way “descended” from humanity and share at least some of the values (love, joy, curiosity, creativity) that I currently hold
Basically, if I sat down with a human from 10,000 years ago, I think there’s a lot we would disagree about, but at the end of the day I think they would get the feeling that I’m an “okay person”. I would like to imagine the same sort of thing holding for whatever follows us.
I don’t find the hair-splitting arguments like “what if the AGI takes over the universe but leaves Earth intact” particularly interesting except insofar as it allows for all 3 of the above. I also don’t think most people have a huge faction of P(~doom) on such weird technicalities.
Well, at least we’ve unearthed the reasons that your p(doom) differs!
Most people do not expect #1(unless we solve alignment), and have a broader definition of #2. I certainly do.