What I do not get is how this disagreement on p(doom) leads to different policy proposals.
If ASI has a 99% probability of killing us all, it is the greatest x-risk we face today and we should obviously be willing to postpone ASI, and possibly singularity (to the extend that in the far future, the diameter of the region of space we colonize at any given time will be a few 100 light years less than what it would be if we focused just on capabilities now).
If ASI has a 1% probability of killing us all, it is still the (debatably) greatest x-risk we face today and we should obviously be willing to postpone ASI etcetera.
To persuade that ASI is safe, one would either not have to care about the far future (for an individual alive today, a 99% of chance of living in a Culture-esque utopia would probably be worth a 1% risk of dying slightly earlier) or provide a much lower p(doom) (e.g. “p(doom)=1e-20, all the x-risk comes from god / the simulators destroying the universe once humans develop ASI, and spending a few centuries on theological research is unlikely to change that” would recommend “just build the damn thing” as a strategy.)
In one sentence, the main reason it matters is because once we drop the assumption of long-termism and impose a limit to how far we care about the future, a 1% probability will give you massive differences in policy compared to a 99% probability of doom, especially if we assume that the benefits and risks are mostly symmetrical. A 1% probability implies that AI should be regulated for tail risks, but a lot of the policies like say a single organization developing AGI or a broad pause become negative EV, under certain other assumptions. 99% obviously flips the script, and massive stoppage of AI, even at the risk of bringing billions of deaths is now positive EV.
And this is worse once we introduce prospect theory, which roughly argues that we overestimate how much we should react to low probability high impact events, because we anchor to misleadingly high probability numbers like 1%, and thus we are likely to massively overestimate the probability of AI doom conditional on the assumption of AI being easy to control being correct.
Strong Evidence is Common generates a way for very low or high probability events to occur, because 1 bit halves the probability conditional on independence.
The relevant thing is how probability both gets clearer and improves with further research enabled by pause. Currently, as a civilization we are at the startled non-sapient deer stage, that’s not a position from which to decide the future of the universe.
The relevant thing is how probability both gets clearer and improves with further research enabled by pause.
I can make the same argument for how probability gets clearer and improves with further research enabled by not pausing, and I actually think this is the case both in general and for this specific problem, so this argument doesn’t work.
Neither this post nor the essay it’s responding to is about policy proposals. So why talk about it? Two points:
As a general principle, if there are two groups who wildly disagree about the facts on the ground, but nevertheless (coincidentally) agree about what policies they favor, then I say they should still probably try to resolve their disagreements if possible, because it’s generally good to have accurate beliefs, e.g. what if both of them are wrong? And maybe that coincidence will not always be true anyway.
It’s not true that the only choice on offer is “Should we ever build ASI? Yes or no?” In fact, that choice (per se) is not on offer at all. What there is, is a gazillion conceivable laws that could be passed, all of which have a wide and idiosyncratic array of intended and unintended consequences. Beyond that, there are a gazillion individual decisions that need to be made, like what careers to pursue, what to donate to, whether to publish or not publish particular things, whether to pursue or not pursue particular lines of research, etc. etc. I find it extraordinarily unlikely that, if Person A thinks p(doom)=99% and Person B thinks p(doom)=1%, then they’re going to agree on all these gazillions of questions. (And empirically, it seems to be clearly not the case that the p(doom)=1% people and the p(doom)=99% people agree on questions of policy.)
What I do not get is how this disagreement on p(doom) leads to different policy proposals.
If ASI has a 99% probability of killing us all, it is the greatest x-risk we face today and we should obviously be willing to postpone ASI, and possibly singularity (to the extend that in the far future, the diameter of the region of space we colonize at any given time will be a few 100 light years less than what it would be if we focused just on capabilities now).
If ASI has a 1% probability of killing us all, it is still the (debatably) greatest x-risk we face today and we should obviously be willing to postpone ASI etcetera.
To persuade that ASI is safe, one would either not have to care about the far future (for an individual alive today, a 99% of chance of living in a Culture-esque utopia would probably be worth a 1% risk of dying slightly earlier) or provide a much lower p(doom) (e.g. “p(doom)=1e-20, all the x-risk comes from god / the simulators destroying the universe once humans develop ASI, and spending a few centuries on theological research is unlikely to change that” would recommend “just build the damn thing” as a strategy.)
In one sentence, the main reason it matters is because once we drop the assumption of long-termism and impose a limit to how far we care about the future, a 1% probability will give you massive differences in policy compared to a 99% probability of doom, especially if we assume that the benefits and risks are mostly symmetrical. A 1% probability implies that AI should be regulated for tail risks, but a lot of the policies like say a single organization developing AGI or a broad pause become negative EV, under certain other assumptions. 99% obviously flips the script, and massive stoppage of AI, even at the risk of bringing billions of deaths is now positive EV.
And this is worse once we introduce prospect theory, which roughly argues that we overestimate how much we should react to low probability high impact events, because we anchor to misleadingly high probability numbers like 1%, and thus we are likely to massively overestimate the probability of AI doom conditional on the assumption of AI being easy to control being correct.
Strong Evidence is Common generates a way for very low or high probability events to occur, because 1 bit halves the probability conditional on independence.
https://www.lesswrong.com/posts/JD7fwtRQ27yc8NoqS/strong-evidence-is-common
The relevant thing is how probability both gets clearer and improves with further research enabled by pause. Currently, as a civilization we are at the startled non-sapient deer stage, that’s not a position from which to decide the future of the universe.
I can make the same argument for how probability gets clearer and improves with further research enabled by not pausing, and I actually think this is the case both in general and for this specific problem, so this argument doesn’t work.
Neither this post nor the essay it’s responding to is about policy proposals. So why talk about it? Two points:
As a general principle, if there are two groups who wildly disagree about the facts on the ground, but nevertheless (coincidentally) agree about what policies they favor, then I say they should still probably try to resolve their disagreements if possible, because it’s generally good to have accurate beliefs, e.g. what if both of them are wrong? And maybe that coincidence will not always be true anyway.
It’s not true that the only choice on offer is “Should we ever build ASI? Yes or no?” In fact, that choice (per se) is not on offer at all. What there is, is a gazillion conceivable laws that could be passed, all of which have a wide and idiosyncratic array of intended and unintended consequences. Beyond that, there are a gazillion individual decisions that need to be made, like what careers to pursue, what to donate to, whether to publish or not publish particular things, whether to pursue or not pursue particular lines of research, etc. etc. I find it extraordinarily unlikely that, if Person A thinks p(doom)=99% and Person B thinks p(doom)=1%, then they’re going to agree on all these gazillions of questions. (And empirically, it seems to be clearly not the case that the p(doom)=1% people and the p(doom)=99% people agree on questions of policy.)