I’d like to support the SIAI but I’m not currently convinced it’s the best-value charity in a dollars-per-life sense, once time-value of money discounting is applied. I’d like to discuss the best non-SIAI charity available.
I respect your intention, and I don’t want to hijack your thread. But if you feel like explaining, I’m interested in the details. Are your concerns about SIAI in particular, or all existential risks charities? For example, the Future of Humanity Institute at Oxford University accepts donations, and uses those donations, in part, to fund further research into the causes of existential risk, and avenues for reducing existential risk. There are also groups aimed at reducing the risk of global nuclear war, and the risk of biotech disasters.
A second question is how to allocate your research efforts. As you note, the huge differences between different charities’ efficiency mean that research into where to donate can have huge value (relative to e.g. spending more hours earning money). Do you plan to do further research as time passes? If so, what endeavors are on your list of “types of philanthropy worth looking into”?
A major problem of many future existential threat charities is evaluating whether they actually are reducing existential risk, or whether they will actually increase it. The evidence of history, for example, indicates that even the best foreign policy experts are not very good at evaluating a policy’s secondary effects and perverse incentives. The result that it is very hard to evaluate whether the net effect of spending money on what is supposed to be “reducing the risk of global thermonuclear war” will actually increase or decrease the risk of global thermonuclear war. The very same multipliers that leverage to massive utility on the assumption of intended consequences leverage to massive disutility on the assumption of perverse consequences.
On the other hand, it’s rather easy to evaluate the net value of a heavily-studied vaccination against an endemic disease, and you can be reasonably certain you’re not actually spreading the very disease you’re trying to fight.
This sounds more like a bias (people want to know they’re successful; ambiguity aversion; want a solid warm glow) than a consideration which could reasonably compete against the economies of scale in existential risk. The penalty for increased uncertainty owing to difficulty of measurement is simply not as large as the orders of magnitude of scope involved.
The orders of magnitude of scope can only matter if you know which way they fall. If a donation to the Nuclear Threat initiative increases the risk of global nuclear war (like a reduction in arsenals deceiving a leader into believing he can make a successful first strike), the orders of magnitude of negative result make it a vastly worse choice than burning a hundred-dollar bill just to see the pretty colors.
If something has a 1% chance of working and a .8% chance of backfiring, the expected utility is the same as what it would be if there was a .2% chance of working, assuming that the benefits and harms are equal and opposite. this has only reduced the expected utility be a factor of 5. Existential risk is so important that it is better than many other things by much more than a factor of 5.
Sure. Now, show me the detailed analysis as to how you got those very precise numbers of your specific proposed intervention in existential risk having a 1.00% chance of working and 0.80% chance of backfiring, instead of the opposite numbers of 0.80% working and 1.00% backfiring.
Because, see, if the odds are the second way, then the expected utility of your intervention is massively, massively negative. Existential risk is so important that while reducing it is better than many other things by much more than a factor of five, increasing it is much, much worse than many evils by much more than a factor of five.
The real universe has no “good intentions” exception, but there’s a cognitive bias which causes people to overestimate the likelihood that an act taken with good intentions will produce the intended good result and underestimate the risks of negative results. When uncorrected for in matters of existential risk, the result could be, because of the mathematics of scope, an unintentional atrocity.
Now, my back-of-the-envelope calculation is that SIAI doesn’t actually increase the risk of an unfriendly AI by actively trying to create friendly AI. There are so many people doing AI anyway, and the default result is so likely to be unfriendly, that SIAI is a decent choice of existential risk charity. If it succeeds, we have upside; if it creates an unfriendly AI, we were screwed anyway.
On the other hand, the Nuclear Threat Initiative is not merely fucking around with what has seemingly been shown to be a fairly stable system in a quest to achieve an endpoint that is itself unlikely to be stable (total nuclear disarmament; the official goal of the NPT), with all sorts of very-hard-to-calculate scenarios which mean it might on net increase risks of nuclear annihilation of humanity. No, it also might be increasing existential threat of, say, runaway greenhouse warming, by secondary discouraging effects on (for example) nuclear power production. There is nobody on the planet who understands human society and economics and power production and everything else involved well enough to say with any confidence whatsoever that a donation to the Nuclear Threat Initiative will have positive expected utility. All we have to go on is the good intentions of the NTI people, which is no more a guarantee than assurances from a local newspaper horoscope that “Your efforts will be rewarded.”
It can never be impossible to determine expected utility because probability is a function of the information that you have. Probability is in the mind; it is part of the map, not the territory.
If you do an expected utility calculation with the little information you know, what you calculate will be the expected utility. If you calculate a higher expected utility for donating then, by the VNM utility theorem, it is what you would prefer. However, it may still not be the right choice because, while donating might be preferable to not donating, learning more about the SIAI might have an even higher expected utility. However, once you have all the relevant information you can get, it is nonsensical say that you can’t calculate the true probability; probability is a quantification of your own knowledge.
Even if you believe that someone else knows something that you don’t, you must make a best guess (well, a best probability distribution of guesses) and make a judgment under uncertainty. People have brought up the possibilities of information being intentionally withheld and of wishful thinking. These are no excuse; account for them to the best of your ability and choose. No matter how unfair the set of evidence you receive is, there is always an optimal probability distribution over what it really means. This is what is used in expected utility calculations.
Yes, that’s what you do. And my analysis is that the best decision under the available uncertainty is that the probability of donating to NTI doing massive good is not distinguishable from the probability of it doing massive harm. The case for 1.0 vs. 0.8 is not any more convincing to me than the case for 0.8 vs. 1.0. Given a hundred questions on the level of whether the Nuclear Threat Initiative is a good thing to do or not, I would not expect my answers to have any more chance of being right than if I answered based entirely on the results of a fair coin. I would, as I said elsewhere in this discussion, take an even-money bet on either side of reality, in the fullness of time, proving the result either is massive weal or massive woe. The massiveness on either side is meaningless because both sides cancel out. The expected utility of a donation to the NTI is, by my estimates, accordingly zero.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size. Anyone opining on whether the NTI is a good or bad idea is, in my opinion, just as foolish as Ptolemy opining on whether the Indian Ocean was enclosed by land in the south. I don’t know, you don’t know, nobody on Earth knows enough to privilege any hypothesis about the value of NTI above any other.
When you don’t know enough to privilege any particular hypothesis over any other, the sheer scale of the possible results doesn’t magically create a reason to act.
I find some of the description of your state of knowledge doubtful.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
Even if there are many false claims of evidence, there could still be some real evidence. If you think that the chance that you could find evidence, which is the conjunction of evidence actually existing and it being findable, isn’t too low, than you could try to search for it. However, from what you said, it seems that this improbability lowers the expected utility enough that it you find it preferable to contribute to other causes. Is that your reasoning? Also, do you think that all this applies to the SIAI?
There is almost certainly real evidence at some level; human beings (and thus human society) are fundamentally deterministic physical systems. I don’t know any method to distinguish the evidence from the noise in the case of, for example, the Nuclear Threat Initiative . . . except handing the problem to a friendly superhuman intelligence. (Which probably will use some method other than the NTI’s to ending the existential threat of global thermonuclear war anyway, rendering such a search for evidence moot.)
It doesn’t apply to the SIAI, because I can’t think of an SIAI high-negative failure mode that isn’t more likely to happen in the absence of the SIAI. The SIAI might make a paperclip maximizer or a sadist . . . but I expect anybody trying to make AIs without taking the explicit care SIAI is using is at least as likely to do so by accident, and I think eventual development of AI is near-certain in the short term (the next thousand years, which against billions of years of existence is certainly the short term). Donations to SIAI accordingly come with an increase in existential threat avoidance (however small and hard-to-estimate the probability), but not an increase in existential threat creation (AI is coming anyway).
(So why haven’t I donated to SIAI? Akrasia. Which isn’t a good thing, but being able to identify it as such in the SIAI donation case at least increases my confidence that my anti-NTI argument isn’t just a rationalization of akrasia in that case.)
We run into the Gambler’s Ruin pretty quickly when dealing with bets concerning existential risk reduction, so the assumption that the benefits and harms are equal and opposite seems questionable. Expected utility calculations need a lot of tweaks in cases like this.
I was not suggesting that this is the actual math; I was merely giving an example to show that the possibility of an existential risk reduction effort backfiring does not necessarily make it a bad idea to contribute.
Say I assign a 0.2% probability to a given intervention averting human extinction. If I assign it a 0.1% probability of bringing about extinction (which otherwise would not have occurred), then I’ve lost half the value of an intervention with a 0.2% probability of success and no risk of backfire. A 0.198% probability of extinction would leave a hundredth of the value.
Even at that point, it seems like quite a stretch to say that the best estimate of the Nuclear Threat Initiative’s existential risk impact is that it is 99% as likely to bring about existential catastrophe as to prevent it. And note that if the risk of backfire is to wipe out more orders of magnitude of x-risk reduction opportunity, the positives and negatives need to be very finely balanced:
If an x-risk reduction intervention has substantially greater probability of averting than producing existential risk, then it’s a win.
If an x-risk reduction intervention has greater probability of producing than averting existential risk, then that means that preventing its use is itself an x-risk intervention with good expected value.
To be neutral, the probability of backfire must be in a very narrow range.
Also, as Anna notes, uncertainty on the numbers at this scale leads to high value of information.
If you assigned a 0.2% probability to a social intervention producing a specific result, I’d mostly be highly skeptical that you have enough, good enough, data to put that precise a number on it. Once probabilities get small enough, they’re too small for the human brain to accurately estimate them.
To be neutral in reality, yes, the probability must be in a very narrow range. To be neutral within the ability of a human brain to evaluate without systematic quantitative study, it just needs to be small enough that you can’t really tell if you’re in case 1 or case 2.
If you assigned a 0.2% probability to a social intervention producing a specific result, I’d mostly be highly skeptical that you have enough, good enough, data to put that precise a number on it. Once probabilities get small enough, they’re too small for the human brain to accurately estimate them.
Do you mean that people tend to be poorly calibrated? You might mean that events or statements to which people assign 0.2% probability happen more often than that. Or you might mean that they happen less often. But either way one should then shift one’s probability estimates to take that information into account.
Or do you mean that such a number would be unstable in response to new information, more thinking about it, getting info on how priming and biases affect estimation (obviously such estimates on the spot depend on noisy factors, and studies show gains just from averaging estimates one makes at different times and so forth), etc?
In either case, if you were compelled to offer betting odds on thousands of independent claims like that (without knowledge of which side of the bet you’d have to take, and otherwise structured to make giving your best estimate the winning strategy) how would you do it?
As an aside, Yvain’s post on probability estimation seems relevant here.
Specifically, I am of the opinion that it is well-demonstrated that calculating adverse consequences of social policy is both sufficiently complicated and sufficiently subject to priming and biases that it is beyond human capacity at this time to accurately estimate whether the well-intentioned efforts of the Nuclear Threat Initiative are more likely to reduce or increase the risk of global thermonuclear war.
If I were forced to take a bet on the issue, I would set the odds at perfectly even. Not because I expect that a full and complete analysis by, say, Omega would come up with the probability being even, but because I have no ability to predict whether Omega would find that the Nuclear Threat initiative reduces or increases the chance of global thermonuclear war.
Currently I don’t think existential risk charities are very appropriate for small-scale individual donations, because of the difficulty of evaluating them. I feel that donating to a long-term research charity is a recipe for either analysis-paralysis or a decision that’s ultimately arbitrary. I’ll definitely continue gathering information, and see whether I can raise my confidence in an existential risk charity enough to consider donating. I think it will take a lot of research.
For any systemic risk charity, you can give a kind of “Drake equation” that arrives at an estimated dollar-per-life based on a sequence of probability estimates. Off the top of my head, I think the global population estimate for 2050 is around 8 billion, assuming the current trend in reducing the number of people in extreme poverty continues (reducing extreme poverty reduces pop. growth). That means you have to arrive at a probability greater than 8,000,000:1 to get a cost-per-life estimate of under $1,000.
At first glance that odds ratio looks pretty generous. But it’s very difficult to have any kind of confidence in the calculation that leads to that. How do I decide between estimates of 10^-3 and 10^-5 likelihood? They’re both too small for me to evaluate informally, and there’s two orders of magnitude difference there. Is there a page where you lay out these estimates? I’ve kind of assumed that this existed, but haven’t seen it yet.
The above calculation seems to be considering only current people, and not much valuing additional years of happy life for current people or better lives relative to current Western standards. Nick Bostrom’s Astronomical Waste paper discusses those issues. Time-discounting isn’t enough to wipe out the effect either, since populations may expand very quickly (e.g. brain emulations/artificial wombs and AI teachers).
Gaverick Matheny’s paper “Reducing the Risk of Human Extinction” is also relevant, although it arbitrarily caps various things (like the rate of population growth) to limit the dominance of the future.
If you care about bringing future people into being, then the expected future population if we avoid existential risk is many, many orders of magnitude greater than the current population of the world and looms very large.
If you don’t care about future people then you have to grapple with the Nonidentity Problem:
Suppose that agents as a community have chosen to deplete rather than conserve certain resources. The consequences of that choice for the persons who exist now and who will come into existence over the next two centuries will be “slightly higher” than under a conservation alternative (Parfit 1987, 362). Thereafter, however, for many centuries the quality of life would be much lower. “The great lowering of the quality of life must provide some moral reason not to choose Depletion” (p. 363). Surely we ought to have chosen conservation in some form or another instead. But at the same time depletion seems to harm no one: while distant future persons, by hypothesis, will suffer the adverse effects of choice of depletion, it is also true that a conservation choice very probably would have changed the timing and manner of the conceptions. Future persons, in other words, owe their suffering but also their very existence to the depletion choice. Provided that that existence is worth having, we seem forced to conclude that depletion does not harm, or make things worse for, and is not otherwise “bad for,” anyone at all (p. 363).
Separately, there seems to be a typo in this paragraph of your post:
Off the top of my head, I think the global population estimate for 2050 is around 8 billion, assuming the current trend in reducing the number of people in extreme poverty continues (reducing extreme poverty reduces pop. growth). That means you have to arrive at a probability greater than 8,000,000:1 to get a cost-per-life estimate of under $1,000.
If you mean “what reduction in the probability of (immediate) extinction is equivalent in expected-lives—of-currently-living-people to saving one life today” then that will be near 1 in 8 billion, not 1 in 8 million. That figure is also a slight underestimate if you only care about curent people because medium-term catastrophes would kill future people who don’t yet exist and many current people may have died by then.
Also, if you’re looking for easier-to-evaluate charities, or bigger higher-status ones endorsed by folk such as Warren Buffett, foreign policy elites, etc, I suggest the Nuclear Threat Initiative as an existence proof of the possibility of spending on x-risk reduction. I wouldn’t recommend giving to it in particular, but it does point to the feasibility of meaningful action. Also see Martin Hellman’s on reducing nuclear risk.
I respect your intention, and I don’t want to hijack your thread. But if you feel like explaining, I’m interested in the details. Are your concerns about SIAI in particular, or all existential risks charities? For example, the Future of Humanity Institute at Oxford University accepts donations, and uses those donations, in part, to fund further research into the causes of existential risk, and avenues for reducing existential risk. There are also groups aimed at reducing the risk of global nuclear war, and the risk of biotech disasters.
A second question is how to allocate your research efforts. As you note, the huge differences between different charities’ efficiency mean that research into where to donate can have huge value (relative to e.g. spending more hours earning money). Do you plan to do further research as time passes? If so, what endeavors are on your list of “types of philanthropy worth looking into”?
A major problem of many future existential threat charities is evaluating whether they actually are reducing existential risk, or whether they will actually increase it. The evidence of history, for example, indicates that even the best foreign policy experts are not very good at evaluating a policy’s secondary effects and perverse incentives. The result that it is very hard to evaluate whether the net effect of spending money on what is supposed to be “reducing the risk of global thermonuclear war” will actually increase or decrease the risk of global thermonuclear war. The very same multipliers that leverage to massive utility on the assumption of intended consequences leverage to massive disutility on the assumption of perverse consequences.
On the other hand, it’s rather easy to evaluate the net value of a heavily-studied vaccination against an endemic disease, and you can be reasonably certain you’re not actually spreading the very disease you’re trying to fight.
This sounds more like a bias (people want to know they’re successful; ambiguity aversion; want a solid warm glow) than a consideration which could reasonably compete against the economies of scale in existential risk. The penalty for increased uncertainty owing to difficulty of measurement is simply not as large as the orders of magnitude of scope involved.
The orders of magnitude of scope can only matter if you know which way they fall. If a donation to the Nuclear Threat initiative increases the risk of global nuclear war (like a reduction in arsenals deceiving a leader into believing he can make a successful first strike), the orders of magnitude of negative result make it a vastly worse choice than burning a hundred-dollar bill just to see the pretty colors.
If something has a 1% chance of working and a .8% chance of backfiring, the expected utility is the same as what it would be if there was a .2% chance of working, assuming that the benefits and harms are equal and opposite. this has only reduced the expected utility be a factor of 5. Existential risk is so important that it is better than many other things by much more than a factor of 5.
Sure. Now, show me the detailed analysis as to how you got those very precise numbers of your specific proposed intervention in existential risk having a 1.00% chance of working and 0.80% chance of backfiring, instead of the opposite numbers of 0.80% working and 1.00% backfiring.
Because, see, if the odds are the second way, then the expected utility of your intervention is massively, massively negative. Existential risk is so important that while reducing it is better than many other things by much more than a factor of five, increasing it is much, much worse than many evils by much more than a factor of five.
The real universe has no “good intentions” exception, but there’s a cognitive bias which causes people to overestimate the likelihood that an act taken with good intentions will produce the intended good result and underestimate the risks of negative results. When uncorrected for in matters of existential risk, the result could be, because of the mathematics of scope, an unintentional atrocity.
Now, my back-of-the-envelope calculation is that SIAI doesn’t actually increase the risk of an unfriendly AI by actively trying to create friendly AI. There are so many people doing AI anyway, and the default result is so likely to be unfriendly, that SIAI is a decent choice of existential risk charity. If it succeeds, we have upside; if it creates an unfriendly AI, we were screwed anyway.
On the other hand, the Nuclear Threat Initiative is not merely fucking around with what has seemingly been shown to be a fairly stable system in a quest to achieve an endpoint that is itself unlikely to be stable (total nuclear disarmament; the official goal of the NPT), with all sorts of very-hard-to-calculate scenarios which mean it might on net increase risks of nuclear annihilation of humanity. No, it also might be increasing existential threat of, say, runaway greenhouse warming, by secondary discouraging effects on (for example) nuclear power production. There is nobody on the planet who understands human society and economics and power production and everything else involved well enough to say with any confidence whatsoever that a donation to the Nuclear Threat Initiative will have positive expected utility. All we have to go on is the good intentions of the NTI people, which is no more a guarantee than assurances from a local newspaper horoscope that “Your efforts will be rewarded.”
It can never be impossible to determine expected utility because probability is a function of the information that you have. Probability is in the mind; it is part of the map, not the territory.
If you do an expected utility calculation with the little information you know, what you calculate will be the expected utility. If you calculate a higher expected utility for donating then, by the VNM utility theorem, it is what you would prefer. However, it may still not be the right choice because, while donating might be preferable to not donating, learning more about the SIAI might have an even higher expected utility. However, once you have all the relevant information you can get, it is nonsensical say that you can’t calculate the true probability; probability is a quantification of your own knowledge.
Even if you believe that someone else knows something that you don’t, you must make a best guess (well, a best probability distribution of guesses) and make a judgment under uncertainty. People have brought up the possibilities of information being intentionally withheld and of wishful thinking. These are no excuse; account for them to the best of your ability and choose. No matter how unfair the set of evidence you receive is, there is always an optimal probability distribution over what it really means. This is what is used in expected utility calculations.
Yes, that’s what you do. And my analysis is that the best decision under the available uncertainty is that the probability of donating to NTI doing massive good is not distinguishable from the probability of it doing massive harm. The case for 1.0 vs. 0.8 is not any more convincing to me than the case for 0.8 vs. 1.0. Given a hundred questions on the level of whether the Nuclear Threat Initiative is a good thing to do or not, I would not expect my answers to have any more chance of being right than if I answered based entirely on the results of a fair coin. I would, as I said elsewhere in this discussion, take an even-money bet on either side of reality, in the fullness of time, proving the result either is massive weal or massive woe. The massiveness on either side is meaningless because both sides cancel out. The expected utility of a donation to the NTI is, by my estimates, accordingly zero.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size. Anyone opining on whether the NTI is a good or bad idea is, in my opinion, just as foolish as Ptolemy opining on whether the Indian Ocean was enclosed by land in the south. I don’t know, you don’t know, nobody on Earth knows enough to privilege any hypothesis about the value of NTI above any other.
When you don’t know enough to privilege any particular hypothesis over any other, the sheer scale of the possible results doesn’t magically create a reason to act.
Your conclusion follows from your premises.
I find some of the description of your state of knowledge doubtful.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
Even if there are many false claims of evidence, there could still be some real evidence. If you think that the chance that you could find evidence, which is the conjunction of evidence actually existing and it being findable, isn’t too low, than you could try to search for it. However, from what you said, it seems that this improbability lowers the expected utility enough that it you find it preferable to contribute to other causes. Is that your reasoning? Also, do you think that all this applies to the SIAI?
There is almost certainly real evidence at some level; human beings (and thus human society) are fundamentally deterministic physical systems. I don’t know any method to distinguish the evidence from the noise in the case of, for example, the Nuclear Threat Initiative . . . except handing the problem to a friendly superhuman intelligence. (Which probably will use some method other than the NTI’s to ending the existential threat of global thermonuclear war anyway, rendering such a search for evidence moot.)
It doesn’t apply to the SIAI, because I can’t think of an SIAI high-negative failure mode that isn’t more likely to happen in the absence of the SIAI. The SIAI might make a paperclip maximizer or a sadist . . . but I expect anybody trying to make AIs without taking the explicit care SIAI is using is at least as likely to do so by accident, and I think eventual development of AI is near-certain in the short term (the next thousand years, which against billions of years of existence is certainly the short term). Donations to SIAI accordingly come with an increase in existential threat avoidance (however small and hard-to-estimate the probability), but not an increase in existential threat creation (AI is coming anyway).
(So why haven’t I donated to SIAI? Akrasia. Which isn’t a good thing, but being able to identify it as such in the SIAI donation case at least increases my confidence that my anti-NTI argument isn’t just a rationalization of akrasia in that case.)
I was thinking more of human-comprehensible evidence when I said `evidence’, but you seem to have found that none of that exists.
I agree with your reasoning about the SIAI.
http://lesswrong.com/lw/3kl/optimizing_fuzzies_and_utilons_the_altruism_chip/ suggests a method for motivating oneself to donate. I haven’t tried this, but the poster found it quite effective.
We run into the Gambler’s Ruin pretty quickly when dealing with bets concerning existential risk reduction, so the assumption that the benefits and harms are equal and opposite seems questionable. Expected utility calculations need a lot of tweaks in cases like this.
I was not suggesting that this is the actual math; I was merely giving an example to show that the possibility of an existential risk reduction effort backfiring does not necessarily make it a bad idea to contribute.
Say I assign a 0.2% probability to a given intervention averting human extinction. If I assign it a 0.1% probability of bringing about extinction (which otherwise would not have occurred), then I’ve lost half the value of an intervention with a 0.2% probability of success and no risk of backfire. A 0.198% probability of extinction would leave a hundredth of the value.
Even at that point, it seems like quite a stretch to say that the best estimate of the Nuclear Threat Initiative’s existential risk impact is that it is 99% as likely to bring about existential catastrophe as to prevent it. And note that if the risk of backfire is to wipe out more orders of magnitude of x-risk reduction opportunity, the positives and negatives need to be very finely balanced:
If an x-risk reduction intervention has substantially greater probability of averting than producing existential risk, then it’s a win.
If an x-risk reduction intervention has greater probability of producing than averting existential risk, then that means that preventing its use is itself an x-risk intervention with good expected value.
To be neutral, the probability of backfire must be in a very narrow range.
Also, as Anna notes, uncertainty on the numbers at this scale leads to high value of information.
If you assigned a 0.2% probability to a social intervention producing a specific result, I’d mostly be highly skeptical that you have enough, good enough, data to put that precise a number on it. Once probabilities get small enough, they’re too small for the human brain to accurately estimate them.
To be neutral in reality, yes, the probability must be in a very narrow range. To be neutral within the ability of a human brain to evaluate without systematic quantitative study, it just needs to be small enough that you can’t really tell if you’re in case 1 or case 2.
Do you mean that people tend to be poorly calibrated? You might mean that events or statements to which people assign 0.2% probability happen more often than that. Or you might mean that they happen less often. But either way one should then shift one’s probability estimates to take that information into account.
Or do you mean that such a number would be unstable in response to new information, more thinking about it, getting info on how priming and biases affect estimation (obviously such estimates on the spot depend on noisy factors, and studies show gains just from averaging estimates one makes at different times and so forth), etc?
In either case, if you were compelled to offer betting odds on thousands of independent claims like that (without knowledge of which side of the bet you’d have to take, and otherwise structured to make giving your best estimate the winning strategy) how would you do it?
As an aside, Yvain’s post on probability estimation seems relevant here.
The second.
Specifically, I am of the opinion that it is well-demonstrated that calculating adverse consequences of social policy is both sufficiently complicated and sufficiently subject to priming and biases that it is beyond human capacity at this time to accurately estimate whether the well-intentioned efforts of the Nuclear Threat Initiative are more likely to reduce or increase the risk of global thermonuclear war.
If I were forced to take a bet on the issue, I would set the odds at perfectly even. Not because I expect that a full and complete analysis by, say, Omega would come up with the probability being even, but because I have no ability to predict whether Omega would find that the Nuclear Threat initiative reduces or increases the chance of global thermonuclear war.
Currently I don’t think existential risk charities are very appropriate for small-scale individual donations, because of the difficulty of evaluating them. I feel that donating to a long-term research charity is a recipe for either analysis-paralysis or a decision that’s ultimately arbitrary. I’ll definitely continue gathering information, and see whether I can raise my confidence in an existential risk charity enough to consider donating. I think it will take a lot of research.
For any systemic risk charity, you can give a kind of “Drake equation” that arrives at an estimated dollar-per-life based on a sequence of probability estimates. Off the top of my head, I think the global population estimate for 2050 is around 8 billion, assuming the current trend in reducing the number of people in extreme poverty continues (reducing extreme poverty reduces pop. growth). That means you have to arrive at a probability greater than 8,000,000:1 to get a cost-per-life estimate of under $1,000.
At first glance that odds ratio looks pretty generous. But it’s very difficult to have any kind of confidence in the calculation that leads to that. How do I decide between estimates of 10^-3 and 10^-5 likelihood? They’re both too small for me to evaluate informally, and there’s two orders of magnitude difference there. Is there a page where you lay out these estimates? I’ve kind of assumed that this existed, but haven’t seen it yet.
The above calculation seems to be considering only current people, and not much valuing additional years of happy life for current people or better lives relative to current Western standards. Nick Bostrom’s Astronomical Waste paper discusses those issues. Time-discounting isn’t enough to wipe out the effect either, since populations may expand very quickly (e.g. brain emulations/artificial wombs and AI teachers).
Gaverick Matheny’s paper “Reducing the Risk of Human Extinction” is also relevant, although it arbitrarily caps various things (like the rate of population growth) to limit the dominance of the future.
If you care about bringing future people into being, then the expected future population if we avoid existential risk is many, many orders of magnitude greater than the current population of the world and looms very large.
If you don’t care about future people then you have to grapple with the Nonidentity Problem:
Separately, there seems to be a typo in this paragraph of your post:
If you mean “what reduction in the probability of (immediate) extinction is equivalent in expected-lives—of-currently-living-people to saving one life today” then that will be near 1 in 8 billion, not 1 in 8 million. That figure is also a slight underestimate if you only care about curent people because medium-term catastrophes would kill future people who don’t yet exist and many current people may have died by then.
Also, if you’re looking for easier-to-evaluate charities, or bigger higher-status ones endorsed by folk such as Warren Buffett, foreign policy elites, etc, I suggest the Nuclear Threat Initiative as an existence proof of the possibility of spending on x-risk reduction. I wouldn’t recommend giving to it in particular, but it does point to the feasibility of meaningful action. Also see Martin Hellman’s on reducing nuclear risk.
Are nukes really an x-risk?