SIAI does not presently exhibit high levels of transparency and accountability… For this reason together with the concerns which I express about Existential Risk and Public Relations, I believe that at present GiveWell’s top ranked charities VillageReach and StopTB are better choices than SIAI
Suppose SIAI were a thousand times less accountable than VillageReach.
Suppose this made SIAI a million times less effective than it could be.
Suppose that even the most efficient Existential Risk charity could only reduce P(uFAI|AGI) by 10^-9
Suppose a negative singularity only set mankind back by 100 years, rather than paperclipping the entire light cone and destroying all human value together.*
A problem with Pascal’s Mugging arguments is that once you commit yourself to taking seriously very unlikely events (because they are multiplied by huge potential utilities), if you want to be consistent, you must take into account all potentially relevant unlikely events, not just the ones that point in your desired direction.
To be sure, you can come up with a story in which SIAI with probability epsilon makes a key positive difference, for bignum expected lives saved. But by the same token you can come up with stories in which SIAI with probability epsilon makes a key negative difference (e.g. by convincing people to abandon fruitful lines of research for fruitless ones), for bignum expected lives lost. Similarly, you can come up with stories in which even a small amount of resources spent elsewhere, with probability epsilon makes a key positive difference (e.g. a child saved from death by potentially curable disease, may grow up to make a critical scientific breakthrough or play a role in preserving world peace), for bignum expected lives saved.
Intuition would have us reject Pascal’s Mugging, but when you think it through in full detail, the logical conclusion is that we should… reject Pascal’s Mugging. It does actually reduce to normality.
Wow. Do you really think this sort of agrument can turn people to SIAI, rather than against any cause that uses tiny probabilities of vast utilities to justify itself?
Seconded, tentatively. I’m afraid that all arguments of this form (for x-risk reduction, Pascal’s Wager, etc.) can’t avoid being rejected out of hand by most people, due to their insatiability: buying even a little bit into such an argument seems to compel a person to spend arbitrary amounts of resources on the improbable thing, and open them to harsh criticism for holding back even a little. That said, a few sincere consequentialists actually will react positively to such arguments, so maybe making them on LW is worthwhile on balance.
Multifoliaterose said his result held even for donors who took Astronomical Waste seriously. This seems unlikely to be the case.
Edit: I didn’t vote you down, but what? SIAI is an Existential Risk charity: the point is to save the entire human race, at low probability. Of course the expected value is going to be calculated by multiplying a tiny probability by an enormous value!
This isn’t true for all existential risks. For example, fears about nuclear war or global warming don’t rely on such tiny probabilities. But discussions of many other risks remind me of this xkcd.
The usual consequentialist case for charity to reduce risks of nuclear war or catastrophic global warming feedbacks does rely on tiny probabilities of donations making a difference. Likwise for voting and for many kinds of scientific and medical research charity.
Edit: not as tiny as in Lark’s comment, although the numbers there are incredibly low.
If the Pascal’s mugging issue is with exerting a small force on a big thing and therefore having a small probability of succeeding, I don’t think that’s even a coherent objection; in a chaotic universe, anything you do may save or destroy individual people and human civilization by redrawing from the distribution, and shifting the distribution of the number of saved lives up by one through e.g. aid or buying someone cryo doesn’t seem fundamentally different from e.g. shifting the distribution of election outcomes. This is more clearly true if you believe in a multiverse such as MWI, but I don’t think it requires that.
ETA: if your distribution on people killed is uniform for say 6 through 10, then definitely saving one person and having a 1 in 5 chance of turning a 10-death situation into a 5-death situation are the same thing except for maybe the identity of the victims, right?
But you didn’t do that. A calculation involves both quantitative and qualitative/structural assumptions, and you slanted all of your qualitative choices toward making your point.
You used a discount rate of 0. (That is, a hypothetical life a hundred generations from now deserves exactly as much of my consideration as people alive today). That totally discredits your calculation.
You used a definition of ‘saving a life’ which suggests that we ought to pay women to get pregnant, have children, and then kill the infant so they are free to get pregnant again. Count that as one life ‘saved’.
You didn’t provide estimates of the opportunity cost in ‘human lives’ of diverting resources toward colonizing the universe. Seems to me those costs could be enormous—in lives at the time of diversion, not in distant-future lives.
Appart from the issue JGWeissman brings up, even if you supposed that the lives saved all occurred 300 years in the future, a reasonable discount rate would still only give you a couple of orders of magnitude.
For example,
1.05^300 = 2.3 * 10^6
Which is nowhere near enough.
Edit: there are discount rates that would give you the result you want, but it still seems pritty plausible that, assuming Astronomical Waste, SIAI isn’t a bad bet.
even if you supposed that the lives saved all occurred 300 years in the future, a reasonable discount rate would still only give you a couple of orders of magnitude.
Sounds about right to me.
For example, 1.05^300 = 2.3 * 10^6
Huh? You think 6 is “a couple”? I wish I had your sex life!
But 5% per annum is far too high. It discounts the next generation to only a quarter of the current generation. Way too steep.
Which is nowhere near enough.
Double huh? And huh? some more. You wrote:
a yearly donation of $1 saves an expected 10^18 lives
You imagine (conservatively) that there are a potential 10^18 lives to save 300 years into the future? Boy, I really wish I had your sex life.
You used a discount rate of 0. (That is, a hypothetical life a hundred generations from now deserves exactly as much of my consideration as people alive today). That totally discredits your calculation.
What makes a life at one time worth more than a life at a different time?
Mathematical well-behavedness may demand this if the universal expansion is not slowing down.
Reciprocity. Future folks aren’t concerned about my wishes, so why should I be concerned about theirs?
What makes a life at one time worth the same as a life at a different time?
In a sense, these are flip answers, because I am not really a utilitarian to begin with. And my rejection of utilitarianism starts by asking how it is possible to sum up utilities for different people. It is adding apples and oranges. There is no natural exchange rate. Utilities are like subjective probabilities of different people—it might make sense to compute a weighted average, but how do you justify your weighting scheme?
I suspect that discussing this topic carefully would take too much of my time from other responsibilities, but I hope this sketch has at least given you some things to think about.
And my rejection of utilitarianism starts by asking how it is possible to sum up utilities for different people. It is adding apples and oranges. There is no natural exchange rate.
Utilities for different people don’t come into it. The question is, how much to you now is a contemporaneous person worth, versus someone in a hundred generations? (Or did you mean utilities of different people?)
You have me confused now. I like apples. Joe likes oranges. Mary wishes to maximize total utility. Should she plant an orange tree or an apple tree?
She should consider the world as it would be if she planted the apple tree, and the world as it would be if she planted the orange tree, and see which one is better as she measures value. (Another option is to plant neither, of course.)
The idea isn’t to add up and maximize everyone’s utility. I agree with you that that makes no sense. The point is, when an agent makes a decision, that agent has to evaluate alternatives, and those alternatives are going to be weighed according to how they score under the agent’s utility function. But utility isn’t just selfish profit. I can value that there be happiness even if I don’t ever get to know about it.
I agree with Kant (perhaps for the only time). Asking how much Joe is worth to Mary is verbotten.
We don’t always have the luxury of not choosing. What should Mary do in a trolley problem where she has to direct a train at either you or Joe (or else you both die)? That said . . .
“Worth” needs to be understood in an expansive sense. Kant is probably right that Mary shouldn’t think, “Now, from whom can I extract more profit for my selfish ends? That’s the one I’ll save.” The things she ought to consider are probably the ones we’re accustomed to thinking of as “selfless”. But she can’t evade making a decision.
The idea isn’t to add up and maximize everyone’s utility. I agree with you that that makes no sense. The point is, when an agent makes a decision, that agent has to evaluate alternatives, and those alternatives are going to be weighed according to how they score under the agent’s utility function. But utility isn’t just selfish profit. I can value that there be happiness even if I don’t ever get to know about it.
Ok, I think we can agree to agree. Revealed preference doesn’t prevent me from incorporating utilitarianish snippets of other peoples utility judgments into my own preferences. I am allowed to be benevolent. But simple math and logic prevent me from doing it all-out, the way that Bentham suggested.
But simple math and logic prevent me from doing it all-out, the way that
Bentham suggested.
What simple math and logic? Utilitarianism seems pretty silly to me too—but adding different people’s utilities together is hardly a show-stopping problem.
The problem I see with utilitarianism is that it is a distant ideal. Ideals of moral behaviour normally work best when they act like a carrot which is slightly out of reach. Utilitarianism conflicts with people’s basic drives. It turns everyone into a sinner.
If you preach utilitarianism, people just think you are trying to manipulate them into giving away all their stuff. Usually that is true—promoters of utilitarianism are usually poorer folk who are after the rich people’s stuff—and have found a moral philosophy that helps them get at it.
Politicians often say they will tax the rich and give the money to the poor. This is because they want the poor people’s votes. Utilitarianism is the ethical equivalent of that. Leaders sometimes publicly promote such policies if they want the support of the masses in order to gain power.
Forbidden is not a fundamental property of the world, it is imposed by theorists with agendas.
Samuelson, Wald, von Neumann, Savage, and the other founders of “revealed preference” forbid us to ask how much Joe (or anything else) is worth, independent of an agent with preferences, such as Mary.
Emmanuel Kant, and anyone else who takes “The categorical imperative” at all seriously, forbids us to ask what Joe is worth to Mary, though we may ask what Joe’s cat Maru is worth to Mary.
I knew I shouldn’t have gotten involved in this thread.
Considered. Not convinced. If that was intended as an argument, then EY was having a very bad day.
He is welcome to his opinion but he is not welcome to substitute his for mine.
The ending was particularly bizarre. It sounded like he was saying that treasury bills don’t pay enough interest to make up for the risk that the US may not be here 300 years from now. But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.
But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.
Eliezer doesn’t disagree: as he says more than once, he’s talking about pure preferences, intrinsic values. Other risks do need to be incorporated, but it seems better to do so directly, rather than through a discounting heuristic. Larks seems to implicitly be doing this with his P(AGI) = 10^-9.
Time travel, the past “still existing”—and utilitariainism? I don’t buy any of that either—but in the context of artificial intelligence, I do agree that building discounting functions into the agent’s ultimate values looks like bad news.
Discounting functions arise because agents don’t know about the future—and can’t predict or control it very well. However, the extent to which they can’t predict or control it is a function of the circumstances and their own capabilities. If you wire temporal discounting into the ultimate preferences of super-Deep Blue—then it can’t ever self-improve to push its prediction horizon further out as it gets more computing power! You are unnecessarily building limitations into it. Better to have no temporal discounting wired in—and let the machine itself figure out to what extent it can predict and control the future—and so figure out the relative value of the present.
The SIAI is actually having a negative effect, by distracting conscientious researchers with all kinds of ridiculous and irrelevant moral issues—which results in the unscrupulous Korean FKU corporation developing machine intelligence first—when otherwise they would have been beaten to it. Their bot promises humans a cure for cancer—but then it gets powerful, eats the internet, and goes on to do all manner of dubious things.
If only you had had the sense to give to VillageReach!
With such low numbers the incidental effects of VillageReach on existential risk (via poverty, population, favoring the efficient philanthropy culture, etc) would dominate. If reasonably favorable, they would mean VillageReach would be a better choice. However, the numbers in the comment are incredibly low.
Suppose SIAI were a thousand times less accountable than VillageReach.
Suppose this made SIAI a million times less effective than it could be.
Suppose that even the most efficient Existential Risk charity could only reduce P(uFAI|AGI) by 10^-9
Suppose the odds of an AI singularity or ‘foom’ were only 10^-9.
Suppose a negative singularity only set mankind back by 100 years, rather than paperclipping the entire light cone and destroying all human value together.*
Even then the expected lives saved by SIAI is ~10^28.
It’s patently obvious that SIAI has an annual income of less than $10^6.
Suppose the marginal dollar is worth 10^3 times less than the average dollar.
Even then a yearly donation of $1 saves an expected10^18 lives.
A yearly donation of £1 to VillageReach saves 100,000,000,000,000,000,000 fewer people.
*Sorry Clippy, but multifoliaterose is damaging you here; SIAI is a lot more amenable to negotiation than anyone else.
A problem with Pascal’s Mugging arguments is that once you commit yourself to taking seriously very unlikely events (because they are multiplied by huge potential utilities), if you want to be consistent, you must take into account all potentially relevant unlikely events, not just the ones that point in your desired direction.
To be sure, you can come up with a story in which SIAI with probability epsilon makes a key positive difference, for bignum expected lives saved. But by the same token you can come up with stories in which SIAI with probability epsilon makes a key negative difference (e.g. by convincing people to abandon fruitful lines of research for fruitless ones), for bignum expected lives lost. Similarly, you can come up with stories in which even a small amount of resources spent elsewhere, with probability epsilon makes a key positive difference (e.g. a child saved from death by potentially curable disease, may grow up to make a critical scientific breakthrough or play a role in preserving world peace), for bignum expected lives saved.
Intuition would have us reject Pascal’s Mugging, but when you think it through in full detail, the logical conclusion is that we should… reject Pascal’s Mugging. It does actually reduce to normality.
Wow. Do you really think this sort of agrument can turn people to SIAI, rather than against any cause that uses tiny probabilities of vast utilities to justify itself?
Seconded, tentatively. I’m afraid that all arguments of this form (for x-risk reduction, Pascal’s Wager, etc.) can’t avoid being rejected out of hand by most people, due to their insatiability: buying even a little bit into such an argument seems to compel a person to spend arbitrary amounts of resources on the improbable thing, and open them to harsh criticism for holding back even a little. That said, a few sincere consequentialists actually will react positively to such arguments, so maybe making them on LW is worthwhile on balance.
Multifoliaterose said his result held even for donors who took Astronomical Waste seriously. This seems unlikely to be the case.
Edit: I didn’t vote you down, but what? SIAI is an Existential Risk charity: the point is to save the entire human race, at low probability. Of course the expected value is going to be calculated by multiplying a tiny probability by an enormous value!
This isn’t true for all existential risks. For example, fears about nuclear war or global warming don’t rely on such tiny probabilities. But discussions of many other risks remind me of this xkcd.
The usual consequentialist case for charity to reduce risks of nuclear war or catastrophic global warming feedbacks does rely on tiny probabilities of donations making a difference. Likwise for voting and for many kinds of scientific and medical research charity.
Edit: not as tiny as in Lark’s comment, although the numbers there are incredibly low.
If the Pascal’s mugging issue is with exerting a small force on a big thing and therefore having a small probability of succeeding, I don’t think that’s even a coherent objection; in a chaotic universe, anything you do may save or destroy individual people and human civilization by redrawing from the distribution, and shifting the distribution of the number of saved lives up by one through e.g. aid or buying someone cryo doesn’t seem fundamentally different from e.g. shifting the distribution of election outcomes. This is more clearly true if you believe in a multiverse such as MWI, but I don’t think it requires that.
ETA: if your distribution on people killed is uniform for say 6 through 10, then definitely saving one person and having a 1 in 5 chance of turning a 10-death situation into a 5-death situation are the same thing except for maybe the identity of the victims, right?
That’s a very different value of “tiny” than the one in Larks’s comment.
This is just Pascal’s mugging again.
If you get to make up the numbers then you can make things turn out however you like.
Not if you deliberately pick numbers orders of magnitude more unfavourable to your point of view than you think is actually the case.
But you didn’t do that. A calculation involves both quantitative and qualitative/structural assumptions, and you slanted all of your qualitative choices toward making your point.
You used a discount rate of 0. (That is, a hypothetical life a hundred generations from now deserves exactly as much of my consideration as people alive today). That totally discredits your calculation.
You used a definition of ‘saving a life’ which suggests that we ought to pay women to get pregnant, have children, and then kill the infant so they are free to get pregnant again. Count that as one life ‘saved’.
You didn’t provide estimates of the opportunity cost in ‘human lives’ of diverting resources toward colonizing the universe. Seems to me those costs could be enormous—in lives at the time of diversion, not in distant-future lives.
Appart from the issue JGWeissman brings up, even if you supposed that the lives saved all occurred 300 years in the future, a reasonable discount rate would still only give you a couple of orders of magnitude.
For example, 1.05^300 = 2.3 * 10^6
Which is nowhere near enough.
Edit: there are discount rates that would give you the result you want, but it still seems pritty plausible that, assuming Astronomical Waste, SIAI isn’t a bad bet.
Sounds about right to me.
Huh? You think 6 is “a couple”? I wish I had your sex life!
But 5% per annum is far too high. It discounts the next generation to only a quarter of the current generation. Way too steep.
Double huh? And huh? some more. You wrote:
You imagine (conservatively) that there are a potential 10^18 lives to save 300 years into the future? Boy, I really wish I had your sex life.
If people 300 years from now are whole brain emulations or AIs, then they could reproduce like software with high population densities.
Alternatively, if the human-size brains were all sucked into the matrix long ago, there may well be about 1 person per-planet.
What makes a life at one time worth more than a life at a different time?
Distance.
Tradition in the field of economics
Mathematical well-behavedness may demand this if the universal expansion is not slowing down.
Reciprocity. Future folks aren’t concerned about my wishes, so why should I be concerned about theirs?
What makes a life at one time worth the same as a life at a different time?
In a sense, these are flip answers, because I am not really a utilitarian to begin with. And my rejection of utilitarianism starts by asking how it is possible to sum up utilities for different people. It is adding apples and oranges. There is no natural exchange rate. Utilities are like subjective probabilities of different people—it might make sense to compute a weighted average, but how do you justify your weighting scheme?
I suspect that discussing this topic carefully would take too much of my time from other responsibilities, but I hope this sketch has at least given you some things to think about.
Utilities for different people don’t come into it. The question is, how much to you now is a contemporaneous person worth, versus someone in a hundred generations? (Or did you mean utilities of different people?)
You have me confused now. I like apples. Joe likes oranges. Mary wishes to maximize total utility. Should she plant an orange tree or an apple tree?
I agree with Kant (perhaps for the only time). Asking how much Joe is worth to Mary is verbotten.
She should consider the world as it would be if she planted the apple tree, and the world as it would be if she planted the orange tree, and see which one is better as she measures value. (Another option is to plant neither, of course.)
The idea isn’t to add up and maximize everyone’s utility. I agree with you that that makes no sense. The point is, when an agent makes a decision, that agent has to evaluate alternatives, and those alternatives are going to be weighed according to how they score under the agent’s utility function. But utility isn’t just selfish profit. I can value that there be happiness even if I don’t ever get to know about it.
We don’t always have the luxury of not choosing. What should Mary do in a trolley problem where she has to direct a train at either you or Joe (or else you both die)? That said . . .
“Worth” needs to be understood in an expansive sense. Kant is probably right that Mary shouldn’t think, “Now, from whom can I extract more profit for my selfish ends? That’s the one I’ll save.” The things she ought to consider are probably the ones we’re accustomed to thinking of as “selfless”. But she can’t evade making a decision.
Ok, I think we can agree to agree. Revealed preference doesn’t prevent me from incorporating utilitarianish snippets of other peoples utility judgments into my own preferences. I am allowed to be benevolent. But simple math and logic prevent me from doing it all-out, the way that Bentham suggested.
Now, which one of us has to tell Eliezer?
What simple math and logic? Utilitarianism seems pretty silly to me too—but adding different people’s utilities together is hardly a show-stopping problem.
The problem I see with utilitarianism is that it is a distant ideal. Ideals of moral behaviour normally work best when they act like a carrot which is slightly out of reach. Utilitarianism conflicts with people’s basic drives. It turns everyone into a sinner.
If you preach utilitarianism, people just think you are trying to manipulate them into giving away all their stuff. Usually that is true—promoters of utilitarianism are usually poorer folk who are after the rich people’s stuff—and have found a moral philosophy that helps them get at it.
Politicians often say they will tax the rich and give the money to the poor. This is because they want the poor people’s votes. Utilitarianism is the ethical equivalent of that. Leaders sometimes publicly promote such policies if they want the support of the masses in order to gain power.
No, what is forbidden is to ask how much Joe is worth in an absolute sense, independent of an agent like Mary.
Utility is not a fundamental property of the world, it is perceived by agents with preferences.
This is rapidly becoming surreal.
Forbidden is not a fundamental property of the world, it is imposed by theorists with agendas.
Samuelson, Wald, von Neumann, Savage, and the other founders of “revealed preference” forbid us to ask how much Joe (or anything else) is worth, independent of an agent with preferences, such as Mary.
Emmanuel Kant, and anyone else who takes “The categorical imperative” at all seriously, forbids us to ask what Joe is worth to Mary, though we may ask what Joe’s cat Maru is worth to Mary.
I knew I shouldn’t have gotten involved in this thread.
Immanuel Kant says we that can’t ask what Joe is worth to Mary?
So what? Why should anyone heed that advice? It is silly.
Utilities of different people, yes. He’s complaining that they don’t add up.
For 2, perhaps consider:
http://lesswrong.com/lw/n2/against_discount_rates/
Considered. Not convinced. If that was intended as an argument, then EY was having a very bad day.
He is welcome to his opinion but he is not welcome to substitute his for mine.
The ending was particularly bizarre. It sounded like he was saying that treasury bills don’t pay enough interest to make up for the risk that the US may not be here 300 years from now. But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.
Eliezer doesn’t disagree: as he says more than once, he’s talking about pure preferences, intrinsic values. Other risks do need to be incorporated, but it seems better to do so directly, rather than through a discounting heuristic. Larks seems to implicitly be doing this with his P(AGI) = 10^-9.
Time travel, the past “still existing”—and utilitariainism? I don’t buy any of that either—but in the context of artificial intelligence, I do agree that building discounting functions into the agent’s ultimate values looks like bad news.
Discounting functions arise because agents don’t know about the future—and can’t predict or control it very well. However, the extent to which they can’t predict or control it is a function of the circumstances and their own capabilities. If you wire temporal discounting into the ultimate preferences of super-Deep Blue—then it can’t ever self-improve to push its prediction horizon further out as it gets more computing power! You are unnecessarily building limitations into it. Better to have no temporal discounting wired in—and let the machine itself figure out to what extent it can predict and control the future—and so figure out the relative value of the present.
Well, in that case, your math is all wrong!
The SIAI is actually having a negative effect, by distracting conscientious researchers with all kinds of ridiculous and irrelevant moral issues—which results in the unscrupulous Korean FKU corporation developing machine intelligence first—when otherwise they would have been beaten to it. Their bot promises humans a cure for cancer—but then it gets powerful, eats the internet, and goes on to do all manner of dubious things.
If only you had had the sense to give to VillageReach!
With such low numbers the incidental effects of VillageReach on existential risk (via poverty, population, favoring the efficient philanthropy culture, etc) would dominate. If reasonably favorable, they would mean VillageReach would be a better choice. However, the numbers in the comment are incredibly low.
We need to be careful with the terminology here. “Saving expected lives” and “expecting to save lives” are pretty different activities.
Thanks & corrected.
Related:
How Much it Matters to Know What Matters: A Back of the Envelope Calculation