Responses on some more minor points (see my previous comment for big-picture responses):
Regarding “BA updates on a point estimate rather than on the full evidence that went into the point estimate”—I don’t understand this claim. BA updates on the full probability distribution of the estimate, which takes into account potential estimate error. The more robust the estimate, the smaller the BA.
Regarding “double-counting” priors, I have not advocated for doing both an explicit “skepticism discount” in one’s EEV calculation and then performing a BA on the output based on the same reasons for skepticism. Instead, I’ve discussed the pros and cons of these two different approaches to accounting for skepticism. There are cases in which I think some sources of skepticism (such as “only 10% of studies in this reference class are replicable”) should be explicitly adjusted for, while others (“If a calculation tells me that an action is the best I can take, I should be skeptical because the conclusion is a priori unlikely”) should be implicitly adjusted for. But I don’t believe anything I’ve said implies that one should “double-count priors.”
Regarding ” log-normal priors would lead to different graphs in the second post, weakening the conclusion. To take the expectation of the logarithm and interpret that as the logarithm of the true cost-effectiveness is to bias the result downward.”—FWIW, I did a version of my original analysis using log-normal distributions (including the correct formula for the expected value) and the picture didn’t change much. I don’t think this issue is an important one though I’m open to being convinced otherwise by detailed analysis.
I don’t find the “charity doomsday argument” compelling. One could believe in low probability of extinction by (a) disputing that our current probability of extinction is high to begin with, or (b) accepting that it’s high but disputing that it can only be lowered by a donation to one of today’s charities (it could be lowered by a large set of diffuse actions, or by a small number of actions whose ability to get funding is overdetermined, or by a far-future charity, or by a combination). If one starts off believing that probability of extinction is high and that it can only be lowered by a particular charity working today that cannot close its funding gap without help from oneself, this seems to beg the question. (I don’t believe this set of propositions.)
I don’t believe any of the alternative solutions to “Pascal’s Mugging” are compelling for all possible constructions of “Pascal’s Mugging.” The only one that seems difficult to get around by modifying the construction is the “bounded utility function” solution, but I don’t believe it is reasonable to have a bounded utility function: I believe, for example, that one should be willing to pay $100 for a 1/N chance of saving N lives for any N>=1, if (as is not the case with “Pascal’s Mugging”) the “1/N chance of saving N lives” calculation is well supported and therefore robust (i.e., has relatively narrow error bars). Thus, “Pascal’s Mugging” remains an example of the sort of “absurd implication” I’d expect for an insufficiently skeptical prior.
Finally, regarding “a single percentage point of reduction of existential risks would be worth (from a utilitarian expected utility point-of-view) a delay of over 10 million years.”—I’m not aware of reasons to believe it’s clear that it would be easier to reduce extinction risk by a percentage point than to speed colonization by 10 million years. If the argument is simply that “a single percentage point seems like a small number,” then I believe this is simply an issue of framing, a case of making something very difficult sound easy by expressing it as a small probability of a fantastically difficult accomplishment. Furthermore, I believe that what you call “speedup” reduces net risk of extinction, so I don’t think the comparison is valid. (I will elaborate on this belief in the future.)
I don’t believe any of the alternative solutions to “Pascal’s Mugging” are compelling for all possible constructions of “Pascal’s Mugging.” The only one that seems difficult to get around by modifying the construction is the “bounded utility function” solution, but I don’t believe it is reasonable to have a bounded utility function: I believe, for example, that one should be willing to pay $100 for a 1/N chance of saving N lives for any N>=1, if (as is not the case with “Pascal’s Mugging”) the “1/N chance of saving N lives” calculation is well supported and therefore robust (i.e., has relatively narrow error bars). Thus, “Pascal’s Mugging” remains an example of the sort of “absurd implication” I’d expect for an insufficiently skeptical prior.
Yes. I would definitely pay significant money to stop e.g. nuclear war conditional on twelve 6-sided dice all rolling 1 . (In the case of dice, pretty much any natural choice of a prior for the initial state of the dice before they bounce results in probability very close to 1⁄6 for each side).
Formally, it is the case that a number which can be postulated in an argument grows faster than any computable function of the length of the argument, if the “argument” is at least Turing complete (i.e. can postulate a Turing machine with a tape for it). And, subsequently, if you base priors on the length alone, the sum is not even well defined, and it’s sign is dependent on the order of summation, and so on.
If we sum in the order of increasing length, everything is dominated by theories that dedicate largest part of their length to making up a really huge number (as even very small increase in this part dramatically boosts the number), so it might even be possible for a super-intelligence or even human-level intelligence to obtain an actionable outcome out of it—something like destroying low temperature labs because the simplest theory which links a very large number to actions does so by modifying laws of physics a little so that very cold liquid helium triggers some sort of world destruction or multiverse destruction, killing people who presumably don’t want to die. Or conversely, liquid helium maximization as it stabilizes some multiverse full of people who’d rather live than die (I’d expect the former to dominate because unusual experiments triggering some sort of instability seems like something that can be postulated more succintly). Or maximization of the number of anti-protons. Something likewise very silly, where the “appeal” is in how much of the theory length it leaves to making the consequences huge. Either way, starting from some good intention (saving people from involuntary death, CEV, or what ever), given a prior that only discounts theories for their length, you don’t get anything particularly nice in the end, you get arbitrarily low (limit of 0) probability of something super good.
Responses on some more minor points (see my previous comment for big-picture responses):
Regarding “BA updates on a point estimate rather than on the full evidence that went into the point estimate”—I don’t understand this claim. BA updates on the full probability distribution of the estimate, which takes into account potential estimate error. The more robust the estimate, the smaller the BA.
Regarding “double-counting” priors, I have not advocated for doing both an explicit “skepticism discount” in one’s EEV calculation and then performing a BA on the output based on the same reasons for skepticism. Instead, I’ve discussed the pros and cons of these two different approaches to accounting for skepticism. There are cases in which I think some sources of skepticism (such as “only 10% of studies in this reference class are replicable”) should be explicitly adjusted for, while others (“If a calculation tells me that an action is the best I can take, I should be skeptical because the conclusion is a priori unlikely”) should be implicitly adjusted for. But I don’t believe anything I’ve said implies that one should “double-count priors.”
Regarding ” log-normal priors would lead to different graphs in the second post, weakening the conclusion. To take the expectation of the logarithm and interpret that as the logarithm of the true cost-effectiveness is to bias the result downward.”—FWIW, I did a version of my original analysis using log-normal distributions (including the correct formula for the expected value) and the picture didn’t change much. I don’t think this issue is an important one though I’m open to being convinced otherwise by detailed analysis.
I don’t find the “charity doomsday argument” compelling. One could believe in low probability of extinction by (a) disputing that our current probability of extinction is high to begin with, or (b) accepting that it’s high but disputing that it can only be lowered by a donation to one of today’s charities (it could be lowered by a large set of diffuse actions, or by a small number of actions whose ability to get funding is overdetermined, or by a far-future charity, or by a combination). If one starts off believing that probability of extinction is high and that it can only be lowered by a particular charity working today that cannot close its funding gap without help from oneself, this seems to beg the question. (I don’t believe this set of propositions.)
I don’t believe any of the alternative solutions to “Pascal’s Mugging” are compelling for all possible constructions of “Pascal’s Mugging.” The only one that seems difficult to get around by modifying the construction is the “bounded utility function” solution, but I don’t believe it is reasonable to have a bounded utility function: I believe, for example, that one should be willing to pay $100 for a 1/N chance of saving N lives for any N>=1, if (as is not the case with “Pascal’s Mugging”) the “1/N chance of saving N lives” calculation is well supported and therefore robust (i.e., has relatively narrow error bars). Thus, “Pascal’s Mugging” remains an example of the sort of “absurd implication” I’d expect for an insufficiently skeptical prior.
Finally, regarding “a single percentage point of reduction of existential risks would be worth (from a utilitarian expected utility point-of-view) a delay of over 10 million years.”—I’m not aware of reasons to believe it’s clear that it would be easier to reduce extinction risk by a percentage point than to speed colonization by 10 million years. If the argument is simply that “a single percentage point seems like a small number,” then I believe this is simply an issue of framing, a case of making something very difficult sound easy by expressing it as a small probability of a fantastically difficult accomplishment. Furthermore, I believe that what you call “speedup” reduces net risk of extinction, so I don’t think the comparison is valid. (I will elaborate on this belief in the future.)
Yes. I would definitely pay significant money to stop e.g. nuclear war conditional on twelve 6-sided dice all rolling 1 . (In the case of dice, pretty much any natural choice of a prior for the initial state of the dice before they bounce results in probability very close to 1⁄6 for each side).
Formally, it is the case that a number which can be postulated in an argument grows faster than any computable function of the length of the argument, if the “argument” is at least Turing complete (i.e. can postulate a Turing machine with a tape for it). And, subsequently, if you base priors on the length alone, the sum is not even well defined, and it’s sign is dependent on the order of summation, and so on.
If we sum in the order of increasing length, everything is dominated by theories that dedicate largest part of their length to making up a really huge number (as even very small increase in this part dramatically boosts the number), so it might even be possible for a super-intelligence or even human-level intelligence to obtain an actionable outcome out of it—something like destroying low temperature labs because the simplest theory which links a very large number to actions does so by modifying laws of physics a little so that very cold liquid helium triggers some sort of world destruction or multiverse destruction, killing people who presumably don’t want to die. Or conversely, liquid helium maximization as it stabilizes some multiverse full of people who’d rather live than die (I’d expect the former to dominate because unusual experiments triggering some sort of instability seems like something that can be postulated more succintly). Or maximization of the number of anti-protons. Something likewise very silly, where the “appeal” is in how much of the theory length it leaves to making the consequences huge. Either way, starting from some good intention (saving people from involuntary death, CEV, or what ever), given a prior that only discounts theories for their length, you don’t get anything particularly nice in the end, you get arbitrarily low (limit of 0) probability of something super good.