It can never be impossible to determine expected utility because probability is a function of the information that you have. Probability is in the mind; it is part of the map, not the territory.
If you do an expected utility calculation with the little information you know, what you calculate will be the expected utility. If you calculate a higher expected utility for donating then, by the VNM utility theorem, it is what you would prefer. However, it may still not be the right choice because, while donating might be preferable to not donating, learning more about the SIAI might have an even higher expected utility. However, once you have all the relevant information you can get, it is nonsensical say that you can’t calculate the true probability; probability is a quantification of your own knowledge.
Even if you believe that someone else knows something that you don’t, you must make a best guess (well, a best probability distribution of guesses) and make a judgment under uncertainty. People have brought up the possibilities of information being intentionally withheld and of wishful thinking. These are no excuse; account for them to the best of your ability and choose. No matter how unfair the set of evidence you receive is, there is always an optimal probability distribution over what it really means. This is what is used in expected utility calculations.
Yes, that’s what you do. And my analysis is that the best decision under the available uncertainty is that the probability of donating to NTI doing massive good is not distinguishable from the probability of it doing massive harm. The case for 1.0 vs. 0.8 is not any more convincing to me than the case for 0.8 vs. 1.0. Given a hundred questions on the level of whether the Nuclear Threat Initiative is a good thing to do or not, I would not expect my answers to have any more chance of being right than if I answered based entirely on the results of a fair coin. I would, as I said elsewhere in this discussion, take an even-money bet on either side of reality, in the fullness of time, proving the result either is massive weal or massive woe. The massiveness on either side is meaningless because both sides cancel out. The expected utility of a donation to the NTI is, by my estimates, accordingly zero.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size. Anyone opining on whether the NTI is a good or bad idea is, in my opinion, just as foolish as Ptolemy opining on whether the Indian Ocean was enclosed by land in the south. I don’t know, you don’t know, nobody on Earth knows enough to privilege any hypothesis about the value of NTI above any other.
When you don’t know enough to privilege any particular hypothesis over any other, the sheer scale of the possible results doesn’t magically create a reason to act.
I find some of the description of your state of knowledge doubtful.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
Even if there are many false claims of evidence, there could still be some real evidence. If you think that the chance that you could find evidence, which is the conjunction of evidence actually existing and it being findable, isn’t too low, than you could try to search for it. However, from what you said, it seems that this improbability lowers the expected utility enough that it you find it preferable to contribute to other causes. Is that your reasoning? Also, do you think that all this applies to the SIAI?
There is almost certainly real evidence at some level; human beings (and thus human society) are fundamentally deterministic physical systems. I don’t know any method to distinguish the evidence from the noise in the case of, for example, the Nuclear Threat Initiative . . . except handing the problem to a friendly superhuman intelligence. (Which probably will use some method other than the NTI’s to ending the existential threat of global thermonuclear war anyway, rendering such a search for evidence moot.)
It doesn’t apply to the SIAI, because I can’t think of an SIAI high-negative failure mode that isn’t more likely to happen in the absence of the SIAI. The SIAI might make a paperclip maximizer or a sadist . . . but I expect anybody trying to make AIs without taking the explicit care SIAI is using is at least as likely to do so by accident, and I think eventual development of AI is near-certain in the short term (the next thousand years, which against billions of years of existence is certainly the short term). Donations to SIAI accordingly come with an increase in existential threat avoidance (however small and hard-to-estimate the probability), but not an increase in existential threat creation (AI is coming anyway).
(So why haven’t I donated to SIAI? Akrasia. Which isn’t a good thing, but being able to identify it as such in the SIAI donation case at least increases my confidence that my anti-NTI argument isn’t just a rationalization of akrasia in that case.)
It can never be impossible to determine expected utility because probability is a function of the information that you have. Probability is in the mind; it is part of the map, not the territory.
If you do an expected utility calculation with the little information you know, what you calculate will be the expected utility. If you calculate a higher expected utility for donating then, by the VNM utility theorem, it is what you would prefer. However, it may still not be the right choice because, while donating might be preferable to not donating, learning more about the SIAI might have an even higher expected utility. However, once you have all the relevant information you can get, it is nonsensical say that you can’t calculate the true probability; probability is a quantification of your own knowledge.
Even if you believe that someone else knows something that you don’t, you must make a best guess (well, a best probability distribution of guesses) and make a judgment under uncertainty. People have brought up the possibilities of information being intentionally withheld and of wishful thinking. These are no excuse; account for them to the best of your ability and choose. No matter how unfair the set of evidence you receive is, there is always an optimal probability distribution over what it really means. This is what is used in expected utility calculations.
Yes, that’s what you do. And my analysis is that the best decision under the available uncertainty is that the probability of donating to NTI doing massive good is not distinguishable from the probability of it doing massive harm. The case for 1.0 vs. 0.8 is not any more convincing to me than the case for 0.8 vs. 1.0. Given a hundred questions on the level of whether the Nuclear Threat Initiative is a good thing to do or not, I would not expect my answers to have any more chance of being right than if I answered based entirely on the results of a fair coin. I would, as I said elsewhere in this discussion, take an even-money bet on either side of reality, in the fullness of time, proving the result either is massive weal or massive woe. The massiveness on either side is meaningless because both sides cancel out. The expected utility of a donation to the NTI is, by my estimates, accordingly zero.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size. Anyone opining on whether the NTI is a good or bad idea is, in my opinion, just as foolish as Ptolemy opining on whether the Indian Ocean was enclosed by land in the south. I don’t know, you don’t know, nobody on Earth knows enough to privilege any hypothesis about the value of NTI above any other.
When you don’t know enough to privilege any particular hypothesis over any other, the sheer scale of the possible results doesn’t magically create a reason to act.
Your conclusion follows from your premises.
I find some of the description of your state of knowledge doubtful.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
Even if there are many false claims of evidence, there could still be some real evidence. If you think that the chance that you could find evidence, which is the conjunction of evidence actually existing and it being findable, isn’t too low, than you could try to search for it. However, from what you said, it seems that this improbability lowers the expected utility enough that it you find it preferable to contribute to other causes. Is that your reasoning? Also, do you think that all this applies to the SIAI?
There is almost certainly real evidence at some level; human beings (and thus human society) are fundamentally deterministic physical systems. I don’t know any method to distinguish the evidence from the noise in the case of, for example, the Nuclear Threat Initiative . . . except handing the problem to a friendly superhuman intelligence. (Which probably will use some method other than the NTI’s to ending the existential threat of global thermonuclear war anyway, rendering such a search for evidence moot.)
It doesn’t apply to the SIAI, because I can’t think of an SIAI high-negative failure mode that isn’t more likely to happen in the absence of the SIAI. The SIAI might make a paperclip maximizer or a sadist . . . but I expect anybody trying to make AIs without taking the explicit care SIAI is using is at least as likely to do so by accident, and I think eventual development of AI is near-certain in the short term (the next thousand years, which against billions of years of existence is certainly the short term). Donations to SIAI accordingly come with an increase in existential threat avoidance (however small and hard-to-estimate the probability), but not an increase in existential threat creation (AI is coming anyway).
(So why haven’t I donated to SIAI? Akrasia. Which isn’t a good thing, but being able to identify it as such in the SIAI donation case at least increases my confidence that my anti-NTI argument isn’t just a rationalization of akrasia in that case.)
I was thinking more of human-comprehensible evidence when I said `evidence’, but you seem to have found that none of that exists.
I agree with your reasoning about the SIAI.
http://lesswrong.com/lw/3kl/optimizing_fuzzies_and_utilons_the_altruism_chip/ suggests a method for motivating oneself to donate. I haven’t tried this, but the poster found it quite effective.