Thanks for this post—I really appreciate the thoughtful discussion of the arguments I’ve made.
I’d like to respond by (a) laying out what I believe is a big-picture point of agreement, which I consider more important than any of the disagreements; (b) responding to what I perceive as the main argument this post makes against the framework I’ve advanced; (c) responding on some more minor points. (c) will be a separate comment due to length constraints.
A big-picture point of agreement: the possibility of vast utility gain does not—in itself—disqualify a giving opportunity as a good one, nor does it establish that the giving opportunity is strong. I’m worried that this point of agreement may be lost on many readers.
The OP makes it sound as though I believe that a high enough EEV is “ruled out” by priors; as discussed below, that is not my position. I agree, and always have, that “Bayesian adjustment does not defeat existential risk charity”; however, I think it defeats an existential risk charity that makes no strong arguments for its ability to make an impact, and relies on a “Pascal’s Mugging” type argument for its appeal.
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding. I don’t believe the OP believes this.
I believe the OP and I are in agreement that one should support an existential risk charity if and only if it makes a strong overall case for its likely impact, a case that goes beyond the observation that even a tiny probability of success would imply high expected value. We may disagree on precisely how high the burden of argumentation is, and we probably disagree on whether MIRI clears that hurdle in its current form, but I don’t believe either of us thinks the burden of argumentation is trivial or is so high that it can never be reached.
Response to what I perceive as the main argument of this post
It seems to me that the main argument of this post runs as follows:
The priors I’m using imply extremely low probabilities for certain events.
We don’t have sufficient reasons to confidently assign such low probabilities to such events.
I think the biggest problems with this argument are as follows:
1 - Most importantly, nothing I’ve written implies an extremely low probability for any particular event. Nick Beckstead’s comment on this post lays out the thinking here. The prior I describe isn’t over expected lives saved or DALYs saved (or a similar metric); it’s over the merit of a proposed action relative to the merits of other possible actions. So if one estimates that action A has a 10^-10 chance of saving 10^30 lives, while action B has a 50% chance of saving 1 life, one could be wrong about the difference between A and B by (a) overestimating the probability that action A will have the intended impact; (b) underestimating the potential impact of action B; (c) leaving out other consequences of A and B; (d) making some other mistake.
My current working theory is that proponents of “Pascal’s Mugging” type arguments tend to neglect the “flow-through effects” of accomplishing good. There are many ways in which helping a person may lead to others’ being helped, and ultimately may lead to a small probability of an enormous impact. Nick Beckstead raises a point similar to this one, and the OP has responded that it’s a new and potentially compelling argument to him. I also think it’s worth bearing in mind that there could be other arguments that we haven’t thought of yet—and because of the structure of the situation, I expect such arguments to be more likely to point to further “regression to the mean” (so to make proponents of “Pascal’s Mugging” arguments less confident that their proposed actions have high relative expected value) than to point in the other direction. This general phenomenon is a major reason that I place less weight on explicit arguments than many in this community—explicit arguments that consist mostly of speculation aren’t very stable or reliable, and when “outside views” point the other way, I expect more explicit reflection to generate more arguments that support the “outside views.”
2 - That said, I don’t accept any of the arguments given here for why it’s unacceptable to assign a very low probability to a proposition. I think there is a general confusion here between “low subjective probability that a proposition is correct” and “high confidence that a proposition isn’t correct”; I don’t think those two things are equivalent. Probabilities are often discussed with an “odds” framing, with the implication that assigning a 10^-10 probability to something means that I’d be willing to wager $10^10 against $1; this framing is a useful thought experiment in many cases, but when the numbers are like this I think it starts encouraging people to confuse their risk aversion with “non-extreme” (i.e., rarely under 1% or over 99%) subjective probabilities. Another framing is to ask, “If we could somehow do a huge number of ‘trials’ of this idea, say by simulating worlds constrained by the observations you’ve made, what would your over/under be for the proportion of trials in which the proposition is true?” and in that case one could simultaneously have an over/under of (10^-10 * # trials) and have extremely low confidence in one’s view.
It seems to me that for any small p, there must be some propositions that we assign a probability at least as small as p. (For example, there must be some X such that the probability of an impact greater than X is smaller than p.) Furthermore, it isn’t the case that assigning small p means that it’s impossible to gather evidence that would change one’s mind about p. For example, if you state to me that you will generate a random integer N1 between 1 and 10^100, there must be some integer N2 that I implicitly assign a probability of <=10^-100 as the output of your exercise. (This is true even if there are substantial “unknown unknowns” involved, for example if I don’t trust that your generator is truly random.) Yet if you complete the exercise and tell me it produced the number N2, I quickly revise my probability from <=10^-100 to over 50%, based on a single quick observation.
For these reasons, I think the argument that “the mere fact that one assigns a sufficiently low probability to a proposition means that one must be in error” would have unacceptable implications and is not supported by the arguments in the OP.
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding
Who? I’m against Pascal’s Mugging. I invented that term to illustrate something that I thought was a fallacy. I’m pretty sure a supermajority of LW would not pay Pascal’s Mugger. I’m on the record as saying that x-risk folk should not argue from low probabilities of large impacts, (1) because there are at least medium-probability interventions against xrisk and these will knock any low-probability interventions off the table if the money used for them is genuinely fungible (admittedly people who donate to anti-asteroid efforts cannot be persuaded to just donate to FAI instead), and (2), with (1) established, that it’s logically rude and bad rationalist form to argue that a probability can be arbitrarily tiny because it makes you insensitive to the state of reality. I can reasonably claim to have personally advanced the art of further refuting Pascal’s Mugging. Who are these mysterious hosts of silly people who believe in Pascal’s Mugging, and what are they doing here of all places?
You can randomly accuse people that what they believe constitutes Pascal’s mugging, but that doesn’t make the accusation a valid argument, unless you show that it’s so.
There’s a very simple test to see if someone actually accepts Pascal’s mugging: Go to them and say “I’ll use my godlike hidden powers to increase your utility by 3^^^3 utilons if you hand over to me the complete contents of your bank account.”
Don’t just claim that something else they believe is the same as Pascal’s mugging or I might equally easily claim that someone buying health insurance is a victim of Pascal’s mugging.
Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t? (In Holden’s comment above, one example in the context of Pascal’s mugging-type problems is a factor of 10^10, even as that’s on the order of the population of the Earth.)
I think any reasonable person hearing “8 lives saved per dollar donated” would file it with Pascal’s mugging (which is Eliezer’s term, but the concept is pretty simple and comprehensible even to someone thinking of less extreme probabilities than Eliezer posits; e.g. Holden, above).
In the linked thread, Rain special-pleads that the topic requires very large numbers to talk about, but jsteinhardt counters that that doesn’t make humans any better at reasoning about tiny probabilities multiplied by large numbers. jsteinhardt also points out that just because you can multiply a small number by a large number doesn’t mean the product actually makes any sense at all.
“Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t?”
No. The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If I said “give me a dollar, and I’ll use my Matrix Lord powers to have three dollars appear in your wallet”, someone can simply respond that the chances of me being a Matrix Lord is less than one in three, so the expected payoff is less than the cost. But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”, even though our intuition says we should (and in this case we trust our intuition).
To put it in brief: Pascal’s Mugging is a interesting problem regarding decision theory which LessWrongers should be hoping to solve (I have an idea towards that direction, which I’m writing a discusion post about, but I’d need mathematicians to tell me if it potentially leads to anything); not just a catchphrase you can use to bash someone else’s calculations when their intuitions differs from yours.
But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties
And so does speed prior.
Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
That seems overly specific. There are many other ways in which priors assigned to highly speculative propositions may not be low enough, or when impact of other available actions on a highly speculative scenario be under-evaluated.
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
To me, Pascal’s Wager is defined by a speculative scenario for which there exist no evidence, which has high enough impact to result in actions which are not based on any evidence, despite the uncertainty towards speculative scenarios.
How THE HELL does the above (ok, I didn’t originally include the second quotation, but still) constitute confusion of Pascal’s Wager and Pascal’s Mugging, let alone “willful misinterpretation” ?
I certainly consider that if you multiply a very tiny probability by a huge payoff and then expect others to take your calculation seriously as a call to action, you’re being silly, however it’s labeled. Humans can’t even consider very tiny probabilities without privileging the hypothesis.
Note also that a crazy mugger could demand $10 or else 10^30 people outside the matrix will die, and then argue that you should rationally trust him 100% so the figure is 10^29 lives/$ , or argue that it is 90% certain that those people will die because he’s a bit uncertain about the danger in the alternate worlds, or the like. It’s not about the probability which mugger estimates, it’s about the probability that the typical payer estimates.
I will certainly admit that the precise label is not my true objection, and apologise if I have seemed to be arguing primarily over definitions (which is of course actually a terrible thing to do in general).
Maybe look at the context of the conversation here? edit: to be specific, you might want to reply to HoldenKarnofsky; after all, the utility of convincing him that he’s incorrect in describing it as “Pascal’s Mugging” type arguments ought to be huge...
edit2: and if it’s not clear, I’m not accusing anyone of anything. Holden said,
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding
I just linked an example of phenomenon which I think may be the cause of Holden’s belief. Feel free to correct him with your brilliant argument that he should simply test if they actually accept Pascal’s Mugging by asking them about 3^^^3 utilons.
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding. I don’t believe the OP believes this.
...
My current working theory is that proponents of “Pascal’s Mugging” type arguments tend to neglect the “flow-through effects” of accomplishing good.
Still might be the thing Holden Karnofsky refers to in the following passages:
And yet remains clearly not the thing that is talked about by either Eliezer or your actual comment.
If it is valuable to make the observation that Holden really isn’t referring to what Eliezer assumes he is then by all means make that point instead of the one you made.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I’ve tried saying this in small letters a number of times, and once in the main post The Pascal’s Wager Fallacy Fallacy, and people apparently just haven’t paid attention, so I’m just going to try shouting it over and over every time somebody makes the same mistake over and over.
edit: And in case it’s not clear, the point is that Pascal’s wager does not depend on the misestimate of probability being low. Any finite variation requires that the probability is high enough .
Likewise, here (linked from the thread I linked) you have both: a prior which is silly high (1 in 2000), and big impact (7 billion lives).
edit: whoops. 1 in 2000 and general talk of low probabilities is in the thread, not in the video. In the video she just goes ahead assigning arbitrary 30% probability to picking an organization with which we live and without which we die, which is obviously so high that much like Pascal’s wager going from 0.5 probability to “the probability could be low, the impact is still infinite!”, so does the LW discussion of this video progress from un-defensible 30% to it doesn’t matter. Let’s picture a Pascal Scam: someone says that there is 50% probability (mostly via ignorance) that unless they are given a lot of money, 10^30 people will die. The audience doesn’t buy 50% probability, but it does still pay up.
(Reply to edit: In the presentation that 30% is one probability in a chain, not an absolute value. Stop with the willful misrepresentations, please.)
From the article:
However, Pascal realizes that the value of 1⁄2 actually plays no real role in the argument, thanks to (2). This brings us to the third, and by far the most important, of his arguments...
If there were a 0.5 probability that the Christian God existed, the wager would make a fuckton more sense. Today we think Pascal’s Wager is a logical fallacy rather than a mere mistaken probability estimate only because later versions of the argument were put forward for lower probabilities, and/or because Pascal went on to argeu that it would carry for lower probabilities.
If the video is where is the actual instance of Pascal’s Wager is being offered in support of SIAI, then it would have been better to link it directly. I also hate video because it’s not searchable, but I can hardly blame you for that, so I will try scanning it.
Before scanning, I precommit to renouncing, abjuring, and distancing MIRI from the argument in the video if it argues for no probability higher than 1 in 2000 of FAI saving the world, because I myself do not positively engage in long-term projects on the basis of probabilities that low (though I sometimes avoid doing things for dangers that small). There ought to be at least one x-risk effort with a greater probability of saving the world than this—or if not, you ought to make one. If you know yourself for an NPC and that you cannot start such a project yourself, you ought to throw money at anyone launching a new project whose probability of saving the world is not known to be this small. 7 billion is also a stupidly low number—x-risk dominates all other optimal philanthropy because of the value of future galaxies, not because of the value of present-day lives. The confluence of these two numbers makes me strongly suspect that, if they are not misquotes in some sense, both low numbers were (presumably unconsciously) chosen to make the ‘lives saved per dollar’ look like a reasonable number in human terms, when in fact the x-risk calculus is such that all utilons should be measured in Probability of OK Outcome because the value of future galaxies stomps everything else.
Attempts to argue for large probabilities that FAI is important, and then tiny probabilities that MIRI is instrumental in creating FAI, will also strike me as a wrongheaded attempt at modesty. On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody’s dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal’s Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal’s Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.
(Scans video.)
This video is primarily about value of information estimates.
“Principle 2: Don’t trust your estimates too much. Estimates in, estimates out.” Good.
Application to the Singularity… It’s explicitly stated that the value is 7 billion lives plus all future generations, which is better—a lower bound is being set, not an estimated exact value.
Final calculation shown:
Probability of eventual AI: 80%
Probability AI with no safeguards will kill us: 80%
(Both of these numbers strike me as a bit suspicious in their apparent medianness which is something that often happens when an argument is unconsciously optimized for sounding reasonable. Really, the probability that AI happens at all, ever, is 80%? Isn’t that a bit low? Is this supposed to be factoring in the probability of nanotechnological warfare wiping out humanity before then, or something? Certainly, AI being possible in principle should have a much more extreme probability than 80%. And a 20% probability of an unsafed AI not killing you sounds like quite an amazing bonanza to get for free. But carrying on...)
Probability we manage safeguards: 40%
(No comment.)
Probability current work is why we manage: 30%
(Arguably too low. Even if MIRI crashes and somebody else carries on successfully, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do twenty years ago.)
Net probabilty: 7%.
This is not necessarily a result I’d agree with, but it’s not a case of Pascal’s Wager on its face. 7% probabilities of large payoffs are a reasonable cause of positive action in sane people; it’s why you would do an Internet startup.
(continues scanning video)
I do not see any slide showing a probability of 1 in 2000. Was this spoken aloud? At what time in the episode?
(Arguably too low. Even if MIRI crashes and somebody else carries on, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do before MIRI.)
It doesn’t merely have to have something to do with MIRI, it must be the case that without funding MIRI we all die, and with funding MIRI, we don’t, and this is precisely the sort of thing that should have very low probability if MIRI is not demonstrably impressive at doing something else.
I do not see any slide showing a probability of 1 in 2000. Was this spoken aloud? At what time in the episode?
Hmm. It is mentioned here and other commenters there likewise talk of low probabilities. I guess I just couldn’t quite imagine someone seriously putting a non small probability on “with MIRI we live, without we die” aspect of it. Startups have quite small probability of success, even without attempting to do the impossible.
edit: And of course what actually matters is donor’s probability.
(Arguably too low. Even if MIRI crashes and somebody else carries on successfully, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do twenty years ago.)
For this to work out to 7%, a donor would need 30% probability that their choice of the organization to donate to is such that with this organization we live, and without, we die.
What donor can be so confident in their choice? Is Thiel this confident? Of course not, he only puts in a small fraction of his income, and he puts more into something like this. By the way I am rather curious about your opinion on this project.
That said, I don’t accept any of the arguments given here for why it’s unacceptable to assign a very low probability to a proposition. I think there is a general confusion here between “low subjective probability that a proposition is correct” and “high confidence that a proposition isn’t correct”; I don’t think those two things are equivalent.
I don’t think you’ve really explained why you don’t accept the arguments in the post. Could you please explain why and how the difference between assigning low probability to something and having high confidence it’s incorrect is relevant? I have several points to discuss, but I need to fully understand your argument before doing so.
And yes, I know I am practicing the dark art of post necromancy. But the discussion has largely been of great quality and I don’t think your comment has been appropriately addressed.
Thanks for this post—I really appreciate the thoughtful discussion of the arguments I’ve made.
I’d like to respond by (a) laying out what I believe is a big-picture point of agreement, which I consider more important than any of the disagreements; (b) responding to what I perceive as the main argument this post makes against the framework I’ve advanced; (c) responding on some more minor points. (c) will be a separate comment due to length constraints.
A big-picture point of agreement: the possibility of vast utility gain does not—in itself—disqualify a giving opportunity as a good one, nor does it establish that the giving opportunity is strong. I’m worried that this point of agreement may be lost on many readers.
The OP makes it sound as though I believe that a high enough EEV is “ruled out” by priors; as discussed below, that is not my position. I agree, and always have, that “Bayesian adjustment does not defeat existential risk charity”; however, I think it defeats an existential risk charity that makes no strong arguments for its ability to make an impact, and relies on a “Pascal’s Mugging” type argument for its appeal.
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding. I don’t believe the OP believes this.
I believe the OP and I are in agreement that one should support an existential risk charity if and only if it makes a strong overall case for its likely impact, a case that goes beyond the observation that even a tiny probability of success would imply high expected value. We may disagree on precisely how high the burden of argumentation is, and we probably disagree on whether MIRI clears that hurdle in its current form, but I don’t believe either of us thinks the burden of argumentation is trivial or is so high that it can never be reached.
Response to what I perceive as the main argument of this post
It seems to me that the main argument of this post runs as follows:
The priors I’m using imply extremely low probabilities for certain events.
We don’t have sufficient reasons to confidently assign such low probabilities to such events.
I think the biggest problems with this argument are as follows:
1 - Most importantly, nothing I’ve written implies an extremely low probability for any particular event. Nick Beckstead’s comment on this post lays out the thinking here. The prior I describe isn’t over expected lives saved or DALYs saved (or a similar metric); it’s over the merit of a proposed action relative to the merits of other possible actions. So if one estimates that action A has a 10^-10 chance of saving 10^30 lives, while action B has a 50% chance of saving 1 life, one could be wrong about the difference between A and B by (a) overestimating the probability that action A will have the intended impact; (b) underestimating the potential impact of action B; (c) leaving out other consequences of A and B; (d) making some other mistake.
My current working theory is that proponents of “Pascal’s Mugging” type arguments tend to neglect the “flow-through effects” of accomplishing good. There are many ways in which helping a person may lead to others’ being helped, and ultimately may lead to a small probability of an enormous impact. Nick Beckstead raises a point similar to this one, and the OP has responded that it’s a new and potentially compelling argument to him. I also think it’s worth bearing in mind that there could be other arguments that we haven’t thought of yet—and because of the structure of the situation, I expect such arguments to be more likely to point to further “regression to the mean” (so to make proponents of “Pascal’s Mugging” arguments less confident that their proposed actions have high relative expected value) than to point in the other direction. This general phenomenon is a major reason that I place less weight on explicit arguments than many in this community—explicit arguments that consist mostly of speculation aren’t very stable or reliable, and when “outside views” point the other way, I expect more explicit reflection to generate more arguments that support the “outside views.”
2 - That said, I don’t accept any of the arguments given here for why it’s unacceptable to assign a very low probability to a proposition. I think there is a general confusion here between “low subjective probability that a proposition is correct” and “high confidence that a proposition isn’t correct”; I don’t think those two things are equivalent. Probabilities are often discussed with an “odds” framing, with the implication that assigning a 10^-10 probability to something means that I’d be willing to wager $10^10 against $1; this framing is a useful thought experiment in many cases, but when the numbers are like this I think it starts encouraging people to confuse their risk aversion with “non-extreme” (i.e., rarely under 1% or over 99%) subjective probabilities. Another framing is to ask, “If we could somehow do a huge number of ‘trials’ of this idea, say by simulating worlds constrained by the observations you’ve made, what would your over/under be for the proportion of trials in which the proposition is true?” and in that case one could simultaneously have an over/under of (10^-10 * # trials) and have extremely low confidence in one’s view.
It seems to me that for any small p, there must be some propositions that we assign a probability at least as small as p. (For example, there must be some X such that the probability of an impact greater than X is smaller than p.) Furthermore, it isn’t the case that assigning small p means that it’s impossible to gather evidence that would change one’s mind about p. For example, if you state to me that you will generate a random integer N1 between 1 and 10^100, there must be some integer N2 that I implicitly assign a probability of <=10^-100 as the output of your exercise. (This is true even if there are substantial “unknown unknowns” involved, for example if I don’t trust that your generator is truly random.) Yet if you complete the exercise and tell me it produced the number N2, I quickly revise my probability from <=10^-100 to over 50%, based on a single quick observation.
For these reasons, I think the argument that “the mere fact that one assigns a sufficiently low probability to a proposition means that one must be in error” would have unacceptable implications and is not supported by the arguments in the OP.
Who? I’m against Pascal’s Mugging. I invented that term to illustrate something that I thought was a fallacy. I’m pretty sure a supermajority of LW would not pay Pascal’s Mugger. I’m on the record as saying that x-risk folk should not argue from low probabilities of large impacts, (1) because there are at least medium-probability interventions against xrisk and these will knock any low-probability interventions off the table if the money used for them is genuinely fungible (admittedly people who donate to anti-asteroid efforts cannot be persuaded to just donate to FAI instead), and (2), with (1) established, that it’s logically rude and bad rationalist form to argue that a probability can be arbitrarily tiny because it makes you insensitive to the state of reality. I can reasonably claim to have personally advanced the art of further refuting Pascal’s Mugging. Who are these mysterious hosts of silly people who believe in Pascal’s Mugging, and what are they doing here of all places?
http://lesswrong.com/lw/6w3/the_125000_summer_singularity_challenge/4krk
You can randomly accuse people that what they believe constitutes Pascal’s mugging, but that doesn’t make the accusation a valid argument, unless you show that it’s so.
There’s a very simple test to see if someone actually accepts Pascal’s mugging: Go to them and say “I’ll use my godlike hidden powers to increase your utility by 3^^^3 utilons if you hand over to me the complete contents of your bank account.”
Don’t just claim that something else they believe is the same as Pascal’s mugging or I might equally easily claim that someone buying health insurance is a victim of Pascal’s mugging.
Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t? (In Holden’s comment above, one example in the context of Pascal’s mugging-type problems is a factor of 10^10, even as that’s on the order of the population of the Earth.)
I think any reasonable person hearing “8 lives saved per dollar donated” would file it with Pascal’s mugging (which is Eliezer’s term, but the concept is pretty simple and comprehensible even to someone thinking of less extreme probabilities than Eliezer posits; e.g. Holden, above).
In the linked thread, Rain special-pleads that the topic requires very large numbers to talk about, but jsteinhardt counters that that doesn’t make humans any better at reasoning about tiny probabilities multiplied by large numbers. jsteinhardt also points out that just because you can multiply a small number by a large number doesn’t mean the product actually makes any sense at all.
No. The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If I said “give me a dollar, and I’ll use my Matrix Lord powers to have three dollars appear in your wallet”, someone can simply respond that the chances of me being a Matrix Lord is less than one in three, so the expected payoff is less than the cost. But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”, even though our intuition says we should (and in this case we trust our intuition).
To put it in brief: Pascal’s Mugging is a interesting problem regarding decision theory which LessWrongers should be hoping to solve (I have an idea towards that direction, which I’m writing a discusion post about, but I’d need mathematicians to tell me if it potentially leads to anything); not just a catchphrase you can use to bash someone else’s calculations when their intuitions differs from yours.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
And so does speed prior.
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Self-consistent isn’t the same as moral.
Bounded utility functions can represent more than your comment suggests, depending on what terms are included. See this discussion.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
Ah, OK, I was thinking of a bounded utility function as one with a “cutoff point”, yes. You’re absolutely right.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
That seems overly specific. There are many other ways in which priors assigned to highly speculative propositions may not be low enough, or when impact of other available actions on a highly speculative scenario be under-evaluated.
To me, Pascal’s Wager is defined by a speculative scenario for which there exist no evidence, which has high enough impact to result in actions which are not based on any evidence, despite the uncertainty towards speculative scenarios.
How THE HELL does the above (ok, I didn’t originally include the second quotation, but still) constitute confusion of Pascal’s Wager and Pascal’s Mugging, let alone “willful misinterpretation” ?
I certainly consider that if you multiply a very tiny probability by a huge payoff and then expect others to take your calculation seriously as a call to action, you’re being silly, however it’s labeled. Humans can’t even consider very tiny probabilities without privileging the hypothesis.
Note also that a crazy mugger could demand $10 or else 10^30 people outside the matrix will die, and then argue that you should rationally trust him 100% so the figure is 10^29 lives/$ , or argue that it is 90% certain that those people will die because he’s a bit uncertain about the danger in the alternate worlds, or the like. It’s not about the probability which mugger estimates, it’s about the probability that the typical payer estimates.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I will certainly admit that the precise label is not my true objection, and apologise if I have seemed to be arguing primarily over definitions (which is of course actually a terrible thing to do in general).
Maybe look at the context of the conversation here? edit: to be specific, you might want to reply to HoldenKarnofsky; after all, the utility of convincing him that he’s incorrect in describing it as “Pascal’s Mugging” type arguments ought to be huge...
edit2: and if it’s not clear, I’m not accusing anyone of anything. Holden said,
I just linked an example of phenomenon which I think may be the cause of Holden’s belief. Feel free to correct him with your brilliant argument that he should simply test if they actually accept Pascal’s Mugging by asking them about 3^^^3 utilons.
Not a Pascal’s Mugging.
Still might be the thing Holden Karnofsky refers to in the following passages:
...
...
And yet remains clearly not the thing that is talked about by either Eliezer or your actual comment.
If it is valuable to make the observation that Holden really isn’t referring to what Eliezer assumes he is then by all means make that point instead of the one you made.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I’ve tried saying this in small letters a number of times, and once in the main post The Pascal’s Wager Fallacy Fallacy, and people apparently just haven’t paid attention, so I’m just going to try shouting it over and over every time somebody makes the same mistake over and over.
In original Pascal’s wager, he had a prior of 0.5 for existence of God.
edit: And in case it’s not clear, the point is that Pascal’s wager does not depend on the misestimate of probability being low. Any finite variation requires that the probability is high enough .
Likewise, here (linked from the thread I linked) you have both: a prior which is silly high (1 in 2000), and big impact (7 billion lives).
edit: whoops. 1 in 2000 and general talk of low probabilities is in the thread, not in the video. In the video she just goes ahead assigning arbitrary 30% probability to picking an organization with which we live and without which we die, which is obviously so high that much like Pascal’s wager going from 0.5 probability to “the probability could be low, the impact is still infinite!”, so does the LW discussion of this video progress from un-defensible 30% to it doesn’t matter. Let’s picture a Pascal Scam: someone says that there is 50% probability (mostly via ignorance) that unless they are given a lot of money, 10^30 people will die. The audience doesn’t buy 50% probability, but it does still pay up.
(Reply to edit: In the presentation that 30% is one probability in a chain, not an absolute value. Stop with the willful misrepresentations, please.)
From the article:
If there were a 0.5 probability that the Christian God existed, the wager would make a fuckton more sense. Today we think Pascal’s Wager is a logical fallacy rather than a mere mistaken probability estimate only because later versions of the argument were put forward for lower probabilities, and/or because Pascal went on to argeu that it would carry for lower probabilities.
If the video is where is the actual instance of Pascal’s Wager is being offered in support of SIAI, then it would have been better to link it directly. I also hate video because it’s not searchable, but I can hardly blame you for that, so I will try scanning it.
Before scanning, I precommit to renouncing, abjuring, and distancing MIRI from the argument in the video if it argues for no probability higher than 1 in 2000 of FAI saving the world, because I myself do not positively engage in long-term projects on the basis of probabilities that low (though I sometimes avoid doing things for dangers that small). There ought to be at least one x-risk effort with a greater probability of saving the world than this—or if not, you ought to make one. If you know yourself for an NPC and that you cannot start such a project yourself, you ought to throw money at anyone launching a new project whose probability of saving the world is not known to be this small. 7 billion is also a stupidly low number—x-risk dominates all other optimal philanthropy because of the value of future galaxies, not because of the value of present-day lives. The confluence of these two numbers makes me strongly suspect that, if they are not misquotes in some sense, both low numbers were (presumably unconsciously) chosen to make the ‘lives saved per dollar’ look like a reasonable number in human terms, when in fact the x-risk calculus is such that all utilons should be measured in Probability of OK Outcome because the value of future galaxies stomps everything else.
Attempts to argue for large probabilities that FAI is important, and then tiny probabilities that MIRI is instrumental in creating FAI, will also strike me as a wrongheaded attempt at modesty. On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody’s dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal’s Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal’s Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.
(Scans video.)
This video is primarily about value of information estimates.
“Principle 2: Don’t trust your estimates too much. Estimates in, estimates out.” Good.
Application to the Singularity… It’s explicitly stated that the value is 7 billion lives plus all future generations, which is better—a lower bound is being set, not an estimated exact value.
Final calculation shown:
Probability of eventual AI: 80%
Probability AI with no safeguards will kill us: 80%
(Both of these numbers strike me as a bit suspicious in their apparent medianness which is something that often happens when an argument is unconsciously optimized for sounding reasonable. Really, the probability that AI happens at all, ever, is 80%? Isn’t that a bit low? Is this supposed to be factoring in the probability of nanotechnological warfare wiping out humanity before then, or something? Certainly, AI being possible in principle should have a much more extreme probability than 80%. And a 20% probability of an unsafed AI not killing you sounds like quite an amazing bonanza to get for free. But carrying on...)
Probability we manage safeguards: 40%
(No comment.)
Probability current work is why we manage: 30%
(Arguably too low. Even if MIRI crashes and somebody else carries on successfully, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do twenty years ago.)
Net probabilty: 7%.
This is not necessarily a result I’d agree with, but it’s not a case of Pascal’s Wager on its face. 7% probabilities of large payoffs are a reasonable cause of positive action in sane people; it’s why you would do an Internet startup.
(continues scanning video)
I do not see any slide showing a probability of 1 in 2000. Was this spoken aloud? At what time in the episode?
It doesn’t merely have to have something to do with MIRI, it must be the case that without funding MIRI we all die, and with funding MIRI, we don’t, and this is precisely the sort of thing that should have very low probability if MIRI is not demonstrably impressive at doing something else.
Hmm. It is mentioned here and other commenters there likewise talk of low probabilities. I guess I just couldn’t quite imagine someone seriously putting a non small probability on “with MIRI we live, without we die” aspect of it. Startups have quite small probability of success, even without attempting to do the impossible.
edit: And of course what actually matters is donor’s probability.
For this to work out to 7%, a donor would need 30% probability that their choice of the organization to donate to is such that with this organization we live, and without, we die.
What donor can be so confident in their choice? Is Thiel this confident? Of course not, he only puts in a small fraction of his income, and he puts more into something like this. By the way I am rather curious about your opinion on this project.
Are you sure they are wrong about what constitutes Pascal’s mugging, rather than about whether the probability of xrisk is low?
I don’t think you’ve really explained why you don’t accept the arguments in the post. Could you please explain why and how the difference between assigning low probability to something and having high confidence it’s incorrect is relevant? I have several points to discuss, but I need to fully understand your argument before doing so.
And yes, I know I am practicing the dark art of post necromancy. But the discussion has largely been of great quality and I don’t think your comment has been appropriately addressed.