I find some of the description of your state of knowledge doubtful.
Furthermore, I am of the opinion that the question is, given the current state of human knowledge, such that no human expert could do better than a fair coin, any more than any Babylonian expert in astronomy could say whether Mars or Sirius was the larger, despite the massive actual difference in their size.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
Even if there are many false claims of evidence, there could still be some real evidence. If you think that the chance that you could find evidence, which is the conjunction of evidence actually existing and it being findable, isn’t too low, than you could try to search for it. However, from what you said, it seems that this improbability lowers the expected utility enough that it you find it preferable to contribute to other causes. Is that your reasoning? Also, do you think that all this applies to the SIAI?
There is almost certainly real evidence at some level; human beings (and thus human society) are fundamentally deterministic physical systems. I don’t know any method to distinguish the evidence from the noise in the case of, for example, the Nuclear Threat Initiative . . . except handing the problem to a friendly superhuman intelligence. (Which probably will use some method other than the NTI’s to ending the existential threat of global thermonuclear war anyway, rendering such a search for evidence moot.)
It doesn’t apply to the SIAI, because I can’t think of an SIAI high-negative failure mode that isn’t more likely to happen in the absence of the SIAI. The SIAI might make a paperclip maximizer or a sadist . . . but I expect anybody trying to make AIs without taking the explicit care SIAI is using is at least as likely to do so by accident, and I think eventual development of AI is near-certain in the short term (the next thousand years, which against billions of years of existence is certainly the short term). Donations to SIAI accordingly come with an increase in existential threat avoidance (however small and hard-to-estimate the probability), but not an increase in existential threat creation (AI is coming anyway).
(So why haven’t I donated to SIAI? Akrasia. Which isn’t a good thing, but being able to identify it as such in the SIAI donation case at least increases my confidence that my anti-NTI argument isn’t just a rationalization of akrasia in that case.)
Your conclusion follows from your premises.
I find some of the description of your state of knowledge doubtful.
50% is a very specific probability. It is reasonable here because it is the prior for the truth of a statement. If there were truly no major pieces of evidence, it could also be your posterior. You may believe that. However, if there are any observations that constitute significant evidence, it is unlikely that they exactly balance out, though it is possible if there is sufficiently little evidence. Given the importance of this, finding out how exactly the pieces of evidence balance would be possible and extremely important in this case.
Yes, if there are any observations that do constitute significant evidence, they are unlikely to balance out. But when a question is of major potential importance, people tend to engage emotionally, which often causes them to take perfectly meaningless noise and interpret it as evidence with significance.
This general cognitive bias to overestimating significance of evidence on issues of importance is an important component of the mind-killing nature of politics. Having misinterpreted noise as evidence, people find it harder to believe that others can honestly evaluate the balance of evidence on such an important issue differently, and find the hypothesis that their opponents are evil more and more plausible, leading to fanaticism.
And, of course, the results of political fanaticism are often disastrous, which means the stakes are high, which means, of course, I may well be being pushed by my emotional reaction to the stakes to overestimate the significance of the evidence that people tend to overestimate the significance of evidence.
Even if there are many false claims of evidence, there could still be some real evidence. If you think that the chance that you could find evidence, which is the conjunction of evidence actually existing and it being findable, isn’t too low, than you could try to search for it. However, from what you said, it seems that this improbability lowers the expected utility enough that it you find it preferable to contribute to other causes. Is that your reasoning? Also, do you think that all this applies to the SIAI?
There is almost certainly real evidence at some level; human beings (and thus human society) are fundamentally deterministic physical systems. I don’t know any method to distinguish the evidence from the noise in the case of, for example, the Nuclear Threat Initiative . . . except handing the problem to a friendly superhuman intelligence. (Which probably will use some method other than the NTI’s to ending the existential threat of global thermonuclear war anyway, rendering such a search for evidence moot.)
It doesn’t apply to the SIAI, because I can’t think of an SIAI high-negative failure mode that isn’t more likely to happen in the absence of the SIAI. The SIAI might make a paperclip maximizer or a sadist . . . but I expect anybody trying to make AIs without taking the explicit care SIAI is using is at least as likely to do so by accident, and I think eventual development of AI is near-certain in the short term (the next thousand years, which against billions of years of existence is certainly the short term). Donations to SIAI accordingly come with an increase in existential threat avoidance (however small and hard-to-estimate the probability), but not an increase in existential threat creation (AI is coming anyway).
(So why haven’t I donated to SIAI? Akrasia. Which isn’t a good thing, but being able to identify it as such in the SIAI donation case at least increases my confidence that my anti-NTI argument isn’t just a rationalization of akrasia in that case.)
I was thinking more of human-comprehensible evidence when I said `evidence’, but you seem to have found that none of that exists.
I agree with your reasoning about the SIAI.
http://lesswrong.com/lw/3kl/optimizing_fuzzies_and_utilons_the_altruism_chip/ suggests a method for motivating oneself to donate. I haven’t tried this, but the poster found it quite effective.