Of possible existential risks, the most feared was a bioengineered pandemic, which got 194 votes (17.8%) - a natural pandemic got 89 (8.2%), making pandemics the overwhelming leader.
This doesn’t look very good from the point of view of the Singularity Institute. While 38.5% of all people have read at least 75% of the Sequences only 16.5% think that unfriendly AI is the most worrisome existential risk.
Is the issue too hard to grasp for most people or has it so far been badly communicated by the Singularity Institute? Or is it simply the wisdom of crowds?
The irony of this is that if, say, 83.5% of respondents instead thought UFAI was the most worrisome existential risk, that would likely be taken as evidence that the LW community was succumbing to groupthink.
My prior belief was that people on less wrong would overestimate the danger of unfriendly ai due to it being part of the reason for Less Wrong’s existence. That probability has decreased since seeing the results, but as I see no reason to believe the opposite would be the case, the effect should still be there.
I don’t quite understand your final clause. Are you saying that you still believe a significant number of people on LW overestimate the danger of UFAI, but that your confidence in that is lower than it was?
More or less. I meant that I now estimate a reduced but still non-zero probability of upwards bias, but only a negligible probability of a bias in the other direction. So the average expected upward bias is decreased but still positive. Thus I should adjust the probability of human extinction being due to unfriendly ai downwards. Of course, the possibility of less wrong over or underestimating existential risk in general is another matter.
The question IIRC wasn’t about the most worrisome, but about the most likely—it is not inconsistent to assign to uFAI (say) 1000 times the disutility of nuclear war but only 0.5 times its probability.
(ETA: I’m assuming worrisomeness is defined as the product of probability times disutility, or a monotonic function thereof.)
I think that worrisomeness should also factor in our ability to do anything about the problem.
If I’m selfish, then I don’t particularly need to worry about global catastrophic risks that will kill (almost) everyone—I’d just die and there’s nothing I can do about it. I’d worry more about risks that are survivable, since they might require some preparation.
If I’m altruistic then I don’t particularly need to worry about risks that are inevitable, or where there is already well-funded and sane mitigation effort going on (since I’d have very little individual ability to make a difference to the probability). I might worry more about risks that have a lower expected disutility but where the mitigation effort is drastically underfunded.
(This is assuming real-world decision theory degenerates into something like CDT; if instead we adopt a more sophisticated decision theory and suppose there are enough other people in our reference class then “selfish” people would behave more like the “altruistic” people in the above paragraph).
Well, if you’re selfish you’d assign more or less the same utility to all states of the world in which you’re dead (unless you believe in afterlife), and in any event you’d assign a higher probability to a particular risk given that “the mitigation effort is drastically underfunded” than given that “there is already well-funded and sane mitigation effort going on”, but you do have a point.
It’s best long-term way, probably. But if you estimate it’ll take 50 years to get a FAI and that some of the existential risks have a significant probability of happening in 10 or 20 years, then you better should try to address them without requiring FAI—or you’re likely to never reach the FAI stage.
In 7 billions of humans, it’s sane to have some individual to focus on FAI now, since it’s a hard problem, so we have to start early; but it’s also normal for not all of us to focus on FAI, but to focus also on other ways to mitigate the existential risks that we estimate are likely to occur before FAI/uFAI.
Hypothetically suppose the following (throughout, assume “AI” stands for significantly superhuman artificial general intelligence):
1) if we fail to develop AI before 2100, various non-AI-related problems kill us all in 2100. 2) if we ever develop unFriendly AI before Friendly AI, UFAI kills us. 3) if we develop FAI before UFAI and before 2100, FAI saves us. 4) FAI isn’t particularly harder to build than UFAI is.
Given those premises, it’s true that UFAI isn’t a major existential risk, in that even if we do nothing about it, UFAI won’t kill us. But it’s also true that FAI is the best (indeed, the only) way to save us.
Are those premises internally contradictory in some way I’m not seeing?
I don’t. Just imagine a hypothetical world where lots of other things are much more certain to kill us much sooner, if we don’t get FAI to solve them soon.
For me the issue with “the most”. Unfriendly AI is a worrisome existential risk, but it still relies on technological breakthrough that we don’t clearly estimate. While “bioengineered pandemic” is something that in the short-term future may very well be possible.
That doesn’t mean SIAI isn’t doing an important job—Friendly AI is a hard task. If you start to try to solve a hard problem when you’re about to die if you don’t, well, it’s too late. So it’s great SIAI people are here to hack away the edges on the problem now.
The phrasing of the question was quite specific: “Which disaster do you think is most likely to wipe out greater than 90% of humanity before the year 2100?”
If I estimate a very small probability of either FAI or UFAI before 2100, then I’m not likely to choose UFAI as “most likely to wipe out 90% of humanity before 2100” if I think there’s a solid chance for something else to do so.
Consider that I interpreted the singularity question to mean “if you think there is any real chance of a singularity, then in the case that the singularity happens, give the year by which you think it has 50% probability.” and answered with 2350, while thinking that the singularity had less than a 50% probability of happening at all.
Yes, Yvain did say to leave it blank if you don’t think there will be a singularity. Given the huge uncertainty involved in anyone’s prediction of the singularity or any question related to it, I took “don’t believe it will happen” to mean that my estimated chance was low enough to not be worth reasoning about the case where it does happen, rather than that my estimate was below 50%.
This doesn’t look very good from the point of view of the Singularity Institute. While 38.5% of all people have read at least 75% of the Sequences only 16.5% think that unfriendly AI is the most worrisome existential risk.
Is the issue too hard to grasp for most people or has it so far been badly communicated by the Singularity Institute? Or is it simply the wisdom of crowds?
The irony of this is that if, say, 83.5% of respondents instead thought UFAI was the most worrisome existential risk, that would likely be taken as evidence that the LW community was succumbing to groupthink.
My prior belief was that people on less wrong would overestimate the danger of unfriendly ai due to it being part of the reason for Less Wrong’s existence. That probability has decreased since seeing the results, but as I see no reason to believe the opposite would be the case, the effect should still be there.
I don’t quite understand your final clause. Are you saying that you still believe a significant number of people on LW overestimate the danger of UFAI, but that your confidence in that is lower than it was?
More or less. I meant that I now estimate a reduced but still non-zero probability of upwards bias, but only a negligible probability of a bias in the other direction. So the average expected upward bias is decreased but still positive. Thus I should adjust the probability of human extinction being due to unfriendly ai downwards. Of course, the possibility of less wrong over or underestimating existential risk in general is another matter.
The question IIRC wasn’t about the most worrisome, but about the most likely—it is not inconsistent to assign to uFAI (say) 1000 times the disutility of nuclear war but only 0.5 times its probability.
(ETA: I’m assuming worrisomeness is defined as the product of probability times disutility, or a monotonic function thereof.)
I think that worrisomeness should also factor in our ability to do anything about the problem.
If I’m selfish, then I don’t particularly need to worry about global catastrophic risks that will kill (almost) everyone—I’d just die and there’s nothing I can do about it. I’d worry more about risks that are survivable, since they might require some preparation.
If I’m altruistic then I don’t particularly need to worry about risks that are inevitable, or where there is already well-funded and sane mitigation effort going on (since I’d have very little individual ability to make a difference to the probability). I might worry more about risks that have a lower expected disutility but where the mitigation effort is drastically underfunded.
(This is assuming real-world decision theory degenerates into something like CDT; if instead we adopt a more sophisticated decision theory and suppose there are enough other people in our reference class then “selfish” people would behave more like the “altruistic” people in the above paragraph).
Well, if you’re selfish you’d assign more or less the same utility to all states of the world in which you’re dead (unless you believe in afterlife), and in any event you’d assign a higher probability to a particular risk given that “the mitigation effort is drastically underfunded” than given that “there is already well-funded and sane mitigation effort going on”, but you do have a point.
The sequences aren’t necessarily claiming UFAI is the single most worrisome risk, just a seriously worrisome risk.
Don’t forget—even if unfriendly AI wasn’t a major existential risk, Friendly AI is still potentially the best way to combat other existential risks.
It’s best long-term way, probably. But if you estimate it’ll take 50 years to get a FAI and that some of the existential risks have a significant probability of happening in 10 or 20 years, then you better should try to address them without requiring FAI—or you’re likely to never reach the FAI stage.
In 7 billions of humans, it’s sane to have some individual to focus on FAI now, since it’s a hard problem, so we have to start early; but it’s also normal for not all of us to focus on FAI, but to focus also on other ways to mitigate the existential risks that we estimate are likely to occur before FAI/uFAI.
How do you imagine a hypothetical world where uFAI is not dangerous enough to kill us, but FAI is powerful enough to save us?
Hypothetically suppose the following (throughout, assume “AI” stands for significantly superhuman artificial general intelligence):
1) if we fail to develop AI before 2100, various non-AI-related problems kill us all in 2100.
2) if we ever develop unFriendly AI before Friendly AI, UFAI kills us.
3) if we develop FAI before UFAI and before 2100, FAI saves us.
4) FAI isn’t particularly harder to build than UFAI is.
Given those premises, it’s true that UFAI isn’t a major existential risk, in that even if we do nothing about it, UFAI won’t kill us. But it’s also true that FAI is the best (indeed, the only) way to save us.
Are those premises internally contradictory in some way I’m not seeing?
No, you’re right. thomblake makes the same point. I just wasn’t thinking carefully enough. Thanks!
I don’t. Just imagine a hypothetical world where lots of other things are much more certain to kill us much sooner, if we don’t get FAI to solve them soon.
More that I think there’s a significant chance that we’re going to get blown up by nukes or a bioweapon before then.
For me the issue with “the most”. Unfriendly AI is a worrisome existential risk, but it still relies on technological breakthrough that we don’t clearly estimate. While “bioengineered pandemic” is something that in the short-term future may very well be possible.
That doesn’t mean SIAI isn’t doing an important job—Friendly AI is a hard task. If you start to try to solve a hard problem when you’re about to die if you don’t, well, it’s too late. So it’s great SIAI people are here to hack away the edges on the problem now.
The phrasing of the question was quite specific: “Which disaster do you think is most likely to wipe out greater than 90% of humanity before the year 2100?”
If I estimate a very small probability of either FAI or UFAI before 2100, then I’m not likely to choose UFAI as “most likely to wipe out 90% of humanity before 2100” if I think there’s a solid chance for something else to do so.
Consider that I interpreted the singularity question to mean “if you think there is any real chance of a singularity, then in the case that the singularity happens, give the year by which you think it has 50% probability.” and answered with 2350, while thinking that the singularity had less than a 50% probability of happening at all.
Yes, Yvain did say to leave it blank if you don’t think there will be a singularity. Given the huge uncertainty involved in anyone’s prediction of the singularity or any question related to it, I took “don’t believe it will happen” to mean that my estimated chance was low enough to not be worth reasoning about the case where it does happen, rather than that my estimate was below 50%.