He’s writing about what the rationalists should do conditional on facing that kind of barbarian. The point of the post is not that rationalist communities should all implement military drafts, but rather that they should be capable of such measures if circumstances require them.
He’s writing about what the rationalists should do conditional on facing that kind of barbarian.
That makes a bit more sense but it still has flaws. The first flaw that comes to mind is that the rationalists may have precommitted to support human rights and that the harm that this precommitment causes to the rationalists in the optimized-barbarian scenario is more than balanced by the benefit it causes by making the rationalists unwilling to violate human rights in other scenarios where the rationalists think they are being attacked by sufficiently optimized barbarians, but are not. Whether this precommitment is rational depends on the probability of the optimized-barbarian scenario, and the fact that it is undeniably harmful conditional on the optimized-barbarian scenario doesn’t mean it’s not rational.
(Edit: It is possible to phrase this in a non-precommitment way: “I know that I am running on corrupted hardware and a threat that seems to be optimized barbarians probably isn’t. The expected utility is calculated from P(false belief in optimized barbarians) benefit from not drafting everyone—P(true belief in optimized barbarians) loss from being killed by barbarians. The conclusion is the same: just because the decision is bad for you in the case where the barbarians really are optimized doesn’t make the choice irrational.)
Right, but it’s a consequentialist argument, not a deontological one. “This measure is more likely to harm than to help” is a good argument; “On the outside view this measure is more likely to harm than to help, and we should go with the outside view even though the inside view seems really really convincing” can be a good argument; but you should never say “This measure would help, but we can’t take it because it’s ‘irrational’”.
In a sense, just saying “you shouldn’t be the kind of ‘rational’ that leads you to be killed by optimized barbarians” is already a consequential argument.
you should never say “This measure would help, but we can’t take it because it’s ‘irrational’”.
Yes, you should, if you have precommitted to not take that measure and you are then unlucky enough to be in the situation where the measure would help. Precommitment means that you can’t change your mind at the last minute and say “now that I know the measure would help, I’ll do it after all”.
The example I give above is not the best illustration because you are precommitting because you can’t distinguish between the two scenarios, but imagine a variation: You always can identify optimized barbarians, but if you precommit to not drafting people when the optimized barbarians come, your government will be trusted more in branches of the world where the optimized barbarians don’t come. Again, the measure of worlds with optimized barbarians is small. In that case, you should precommit, and then if the optimized barbarians come, you say “drafting people would help, but I can’t break a precommitment”. If you have made the precommitment by self-modifying so as to be unwilling to take the measure, the unwillingness looks like a failure of rationality (“If only you weren’t unwilling, you’d survive, but since you are, you’ll die!”), when it really isn’t.
Precommitment is also a potentially good reason. I’m not sure what we disagree about anymore.
Is your objection to the Barbarians post that you fear it will be used to justify actually implementing unsavory measures like those it describes for use against barbarians?
If there is precommitment, it may be true that doing X would benefit you, you refuse to do X, and your refusal to do X is rational.
Eliezer said that if doing X would benefit you and you refuse to do X, that is not really rational.
Furthermore, whether X is rational depends on what the probability of the scenario was before it happened—even if the scenario is happening now. Eliezer as interpreted by you believes that if the scenario is happening now, the past probability of the scenario doesn’t affect whether X is rational (That’s why he could use optimized barbarians as an example in the first place, despite its low probability.)
Also, I happen to think that many cases of people “irrationally” acting on principle can be modelled as a type of precommitment. Precommitment is just how we formalize “I’m going to shop at the store with the lower price, even if I have to spend more on gas to get there” or “we should allow free speech/free press/etc. and I don’t care how many terrorists that helps”.
He’s writing about what the rationalists should do conditional on facing that kind of barbarian. The point of the post is not that rationalist communities should all implement military drafts, but rather that they should be capable of such measures if circumstances require them.
That makes a bit more sense but it still has flaws. The first flaw that comes to mind is that the rationalists may have precommitted to support human rights and that the harm that this precommitment causes to the rationalists in the optimized-barbarian scenario is more than balanced by the benefit it causes by making the rationalists unwilling to violate human rights in other scenarios where the rationalists think they are being attacked by sufficiently optimized barbarians, but are not. Whether this precommitment is rational depends on the probability of the optimized-barbarian scenario, and the fact that it is undeniably harmful conditional on the optimized-barbarian scenario doesn’t mean it’s not rational.
(Edit: It is possible to phrase this in a non-precommitment way: “I know that I am running on corrupted hardware and a threat that seems to be optimized barbarians probably isn’t. The expected utility is calculated from P(false belief in optimized barbarians) benefit from not drafting everyone—P(true belief in optimized barbarians) loss from being killed by barbarians. The conclusion is the same: just because the decision is bad for you in the case where the barbarians really are optimized doesn’t make the choice irrational.)
Right, but it’s a consequentialist argument, not a deontological one. “This measure is more likely to harm than to help” is a good argument; “On the outside view this measure is more likely to harm than to help, and we should go with the outside view even though the inside view seems really really convincing” can be a good argument; but you should never say “This measure would help, but we can’t take it because it’s ‘irrational’”.
In a sense, just saying “you shouldn’t be the kind of ‘rational’ that leads you to be killed by optimized barbarians” is already a consequential argument.
Yes, you should, if you have precommitted to not take that measure and you are then unlucky enough to be in the situation where the measure would help. Precommitment means that you can’t change your mind at the last minute and say “now that I know the measure would help, I’ll do it after all”.
The example I give above is not the best illustration because you are precommitting because you can’t distinguish between the two scenarios, but imagine a variation: You always can identify optimized barbarians, but if you precommit to not drafting people when the optimized barbarians come, your government will be trusted more in branches of the world where the optimized barbarians don’t come. Again, the measure of worlds with optimized barbarians is small. In that case, you should precommit, and then if the optimized barbarians come, you say “drafting people would help, but I can’t break a precommitment”. If you have made the precommitment by self-modifying so as to be unwilling to take the measure, the unwillingness looks like a failure of rationality (“If only you weren’t unwilling, you’d survive, but since you are, you’ll die!”), when it really isn’t.
Precommitment is also a potentially good reason. I’m not sure what we disagree about anymore.
Is your objection to the Barbarians post that you fear it will be used to justify actually implementing unsavory measures like those it describes for use against barbarians?
If there is precommitment, it may be true that doing X would benefit you, you refuse to do X, and your refusal to do X is rational.
Eliezer said that if doing X would benefit you and you refuse to do X, that is not really rational.
Furthermore, whether X is rational depends on what the probability of the scenario was before it happened—even if the scenario is happening now. Eliezer as interpreted by you believes that if the scenario is happening now, the past probability of the scenario doesn’t affect whether X is rational (That’s why he could use optimized barbarians as an example in the first place, despite its low probability.)
Also, I happen to think that many cases of people “irrationally” acting on principle can be modelled as a type of precommitment. Precommitment is just how we formalize “I’m going to shop at the store with the lower price, even if I have to spend more on gas to get there” or “we should allow free speech/free press/etc. and I don’t care how many terrorists that helps”.
TDT/UDT and the outside view are how we formalize precommitment.