Yes, I read the whole thread (and the banned doubleplus ungood post by Roko).
I wouldn’t mind if you were putting “your money” on the table. What I mind is threatening to take act with the goal of reducing mankind’s chances of survival. That’s not “your money”.
If you had just threatened to stop donating money to SIAI (do you donate?), no problemo. Whether that action has an impact on existential risk is unclear; my problem isn’t doing actions that might have increase existential risk, it’s doing actions whose purpose is to increase existential risk. Or even brainstorming about those.
Imagine that Omega guy came down from the sky and gave each human on earth a device with a button. The button can only be pushed once, and has a one-in-a-million chance of making the sun go Nova (and the button only works for the rightful owner).
What would you think of someone who publicly threatened to push his device’s button If the US elected the wrong president? If the government didn’t give him $500 a month? If some website didn’t adopt the moderation policy he prefered? Or someone brainstorming about how to build a device like that, with the above uses in mind?
What if somebody else had a similar button, but with 1 in 100,000 probability. Would it be ok to threaten to push your 1 in a million button if the other guy pressed his 1 in 100,000 button? If you had reason to believe that the other guy, for some reason, would take your threat seriously, but wasn’t taking the threat of his own button seriously?
OK, that got kind of convoluted, but do you see what I’m saying?
If you were sufficiently certain that the situation is as you describe (least convenient world), yes, it would be OK to threaten (and carry out the threat). If however you obtained the information through a bug-ridden device that is known to be biased towards overconfidence in this kind of situations—then such threats would be immoral. And I think most imaginable real-world situations fall in the second category.
I think the only thing we disagree on then is the word “immoral”. I would say that it may very well be incorrect, but not immoral, so long as he is being sincere in his motivations.
ETA: ok after thinking about it some more, I guess I could see how it might be considered immoral (in the least convenient world to the point of view I’m arguing). I guess it kind of depends on the specifics of what’s going on inside his head, which I’m of course not privy to.
I’m not sure it would count as “immoral”, guess it also depends of how you define the terms.
I see this as a case of the more general “does the end justify the means?”. In principle, the end do justify the means, if you’re sufficiently confident that those means will indeed result in that end, and that the end is really valuable. In practice, the human brain is biased towards finding ends to justify means that just happen to bring power or prestige to the human in question. In fact, most people doing anything widely considered “bad” can come up with a good story about how it’s kinda justified.
So, part of what makes a human moral is willingness to correct this, to listen to the voice of doubt, or at least to consider that one may be wrong, especially when taking action that might harm others.
Yes, I read the whole thread (and the banned doubleplus ungood post by Roko).
I wouldn’t mind if you were putting “your money” on the table. What I mind is threatening to take act with the goal of reducing mankind’s chances of survival. That’s not “your money”.
If you had just threatened to stop donating money to SIAI (do you donate?), no problemo. Whether that action has an impact on existential risk is unclear; my problem isn’t doing actions that might have increase existential risk, it’s doing actions whose purpose is to increase existential risk. Or even brainstorming about those.
Imagine that Omega guy came down from the sky and gave each human on earth a device with a button. The button can only be pushed once, and has a one-in-a-million chance of making the sun go Nova (and the button only works for the rightful owner).
What would you think of someone who publicly threatened to push his device’s button If the US elected the wrong president? If the government didn’t give him $500 a month? If some website didn’t adopt the moderation policy he prefered? Or someone brainstorming about how to build a device like that, with the above uses in mind?
What if somebody else had a similar button, but with 1 in 100,000 probability. Would it be ok to threaten to push your 1 in a million button if the other guy pressed his 1 in 100,000 button? If you had reason to believe that the other guy, for some reason, would take your threat seriously, but wasn’t taking the threat of his own button seriously?
OK, that got kind of convoluted, but do you see what I’m saying?
If you were sufficiently certain that the situation is as you describe (least convenient world), yes, it would be OK to threaten (and carry out the threat). If however you obtained the information through a bug-ridden device that is known to be biased towards overconfidence in this kind of situations—then such threats would be immoral. And I think most imaginable real-world situations fall in the second category.
I think the only thing we disagree on then is the word “immoral”. I would say that it may very well be incorrect, but not immoral, so long as he is being sincere in his motivations.
ETA: ok after thinking about it some more, I guess I could see how it might be considered immoral (in the least convenient world to the point of view I’m arguing). I guess it kind of depends on the specifics of what’s going on inside his head, which I’m of course not privy to.
I’m not sure it would count as “immoral”, guess it also depends of how you define the terms.
I see this as a case of the more general “does the end justify the means?”. In principle, the end do justify the means, if you’re sufficiently confident that those means will indeed result in that end, and that the end is really valuable. In practice, the human brain is biased towards finding ends to justify means that just happen to bring power or prestige to the human in question. In fact, most people doing anything widely considered “bad” can come up with a good story about how it’s kinda justified.
So, part of what makes a human moral is willingness to correct this, to listen to the voice of doubt, or at least to consider that one may be wrong, especially when taking action that might harm others.