After reading your comments, here’s my current explanation of what’s up with the bomb argument:
The problem with claiming that taking Left is wrong in the bomb-problem, is in the inference that “because I see the bomb is in the left box” this necessarily implies that “I am in the one-in-a-trillion-trillion situation where the predictor was inaccurate”.
However, this is forgetting the other option, where this setup is one of the vastly larger other worlds where I am being predicted by the predictor. In those worlds, it’s important that I make the worlds logically inconsistent by taking the left box, and so the predictor cannot accurately predict me taking the right box. Otherwise it may notice that this is a fixed point (i.e. leaving a note saying he predicted I’d take the right box does in fact lead to me taking the right box) and then call it a day and execute an action (that I don’t want) with an accurate prediction.
So there’s a 1-in-a-trillion-trillion chance that I am in the real game, and a notably higher chance that I’m in the mind of some agent predicting what I’ll do here. (And in that simulation-world, I don’t want to do the thing I don’t want them to predict that I’ll do.)
So make your estimates of the probably you’re being perfectly simulated and the probability you’re in the real game, and then compare them, multiply by utilities, and you’re done.
Then I’m a bit confused about how to estimate that probability, but I suspect the reasoning goes like this:
It’s near-certain that I will be simulated in some scenarios by the predictor, and it’s one-in-a-trillion-trillions that this is the real scenario. This scenario seems “plausible” as a scenario the predictor would simulate me in, especially given that the alternative ground-truth is that this is the scenario that they actually went with! I’m going to put it somewhere between “10%” and “100%”. So I think the odds ratio is around the order of magnitude of “one-in-ten” to “one-in-a-trillion-trillion”. And when I multiply them by the expected utility, the ratio is still well in favor of taking the bomb and making it very likely that in reality I will not lose 100 dollars.
Sanity check
As a sanity-check, I note this implies that if the utilities-times-probabilities are different, I would not mind taking the $100 hit. Let’s see what the math says here, and then check whether my intuitions agree.
Suppose I value my life at $1 million. Then I think that I should become more indifferent here when the probability of a mistaken simulation approaches 1 in 100,000, or where the money on the line is closer to $10−17.
[You can skip this, but here’s me stating the two multiplications I compared:
World 1: I fake-kill myself to save $X, with probability 110
World 2: I actually kill myself (cost: $1MM), with probability 1Y
To find the indifference point I want the two multiplications of utility-to-probability to come out to be equal. If X = $100, then Y equals 100,000. If Y is a trillion trillion (1024), then X = 10−17. (Unless I did the math wrong.)]
I think this doesn’t obviously clash with my intuitions, and somewhat matches them.
If the simulator was getting things wrong 1 in 100,000 times, I think I’d be more careful with my life in the “real world case” (insofar as that is a sensible concept). Going further, if you told me they were wrong 1 in 10 times, this would change my action, so there’s got to be a tipping point somewhere, and this seems reasonable for many people (though I actually value my life at more than $1MM).
And if the money was that tiny ($10−17), I’d be fairly open to “not taking even the one-in-a-trillion-trillion chance”. (Though really my intuition is that I don’t care about money way before $10^-17, and would probably not risk anything serious starting at like 0.1 cents, because that sort of money seems kind of irritating to have to deal with. So my intuition doesn’t match perfectly here. Though I think that if I were expecting to play trillions of such games, then I would start to actively care about such tiny amounts of money.)
Thanks, this comment thread was pretty helpful.
After reading your comments, here’s my current explanation of what’s up with the bomb argument:
Then I’m a bit confused about how to estimate that probability, but I suspect the reasoning goes like this:
Sanity check
As a sanity-check, I note this implies that if the utilities-times-probabilities are different, I would not mind taking the $100 hit. Let’s see what the math says here, and then check whether my intuitions agree.
Suppose I value my life at $1 million. Then I think that I should become more indifferent here when the probability of a mistaken simulation approaches 1 in 100,000, or where the money on the line is closer to $10−17.
[You can skip this, but here’s me stating the two multiplications I compared:
World 1: I fake-kill myself to save $X, with probability 110
World 2: I actually kill myself (cost: $1MM), with probability 1Y
To find the indifference point I want the two multiplications of utility-to-probability to come out to be equal. If X = $100, then Y equals 100,000. If Y is a trillion trillion (1024), then X = 10−17. (Unless I did the math wrong.)]
I think this doesn’t obviously clash with my intuitions, and somewhat matches them.
If the simulator was getting things wrong 1 in 100,000 times, I think I’d be more careful with my life in the “real world case” (insofar as that is a sensible concept). Going further, if you told me they were wrong 1 in 10 times, this would change my action, so there’s got to be a tipping point somewhere, and this seems reasonable for many people (though I actually value my life at more than $1MM).
And if the money was that tiny ($10−17), I’d be fairly open to “not taking even the one-in-a-trillion-trillion chance”. (Though really my intuition is that I don’t care about money way before $10^-17, and would probably not risk anything serious starting at like 0.1 cents, because that sort of money seems kind of irritating to have to deal with. So my intuition doesn’t match perfectly here. Though I think that if I were expecting to play trillions of such games, then I would start to actively care about such tiny amounts of money.)