Let’s consider two scenarios S1 and S2, with S1 having a lesser harm and S2 a greater harm. M1 and M2 are the events where a mugger presents S1 and S2 respectively. Our bayesian update for either takes the form
Your argument is that (edit: maybe not; I’m going to leave this comment here anyway, hope that’s OK)
)
is greater than
)
and this increase offsets increases in the utility part of the expected utility equation. I’m pretty tired right now, but it seems possible that you could do some algebra and determine what the shape of the function determining this conditional probability as a function of the harms amount would have to be in order to achieve this. In any case, while it might be convenient for your updating module to have the property that this particular function possesses this particular shape, it’s not obvious that it’s the correct way to update. (And in particular, if you were programming an AI and you gave it what you thought was a reasonably good Bayesian updating module, it’s not obvious that the module would update the way you would like on this particular problem, or that you could come up with a sufficiently rigorous description of the scenarios under which it should abandon its regular updating module in order to update in the way you would like.)
Even if this argument works, I’m not sure it completely solves the problem, because it seems possible that there are scenarios with extreme harms and relatively high (but still extremely miniscule) prior probabilities. I don’t know how we would feel about our ideal expected utility maximizer assigning most of its weight to some runaway hypothetical.
Another thought: if a mugger had read your post and understood your argument, perhaps they would choose to mug for smaller amounts. So would that bring
) above
)? What if the mugger can show you their source code and prove that for them the two probabilities are equal? (E.g. they do occasionally play this mugging strategy and when they do they do it with equal probability for any of three harm amounts, all astronomically large.)
Let’s consider two scenarios S1 and S2, with S1 having a lesser harm and S2 a greater harm. M1 and M2 are the events where a mugger presents S1 and S2 respectively. Our bayesian update for either takes the form
Your argument is that (edit: maybe not; I’m going to leave this comment here anyway, hope that’s OK)
is greater than
and this increase offsets increases in the utility part of the expected utility equation. I’m pretty tired right now, but it seems possible that you could do some algebra and determine what the shape of the function determining this conditional probability as a function of the harms amount would have to be in order to achieve this. In any case, while it might be convenient for your updating module to have the property that this particular function possesses this particular shape, it’s not obvious that it’s the correct way to update. (And in particular, if you were programming an AI and you gave it what you thought was a reasonably good Bayesian updating module, it’s not obvious that the module would update the way you would like on this particular problem, or that you could come up with a sufficiently rigorous description of the scenarios under which it should abandon its regular updating module in order to update in the way you would like.)
Even if this argument works, I’m not sure it completely solves the problem, because it seems possible that there are scenarios with extreme harms and relatively high (but still extremely miniscule) prior probabilities. I don’t know how we would feel about our ideal expected utility maximizer assigning most of its weight to some runaway hypothetical.
Another thought: if a mugger had read your post and understood your argument, perhaps they would choose to mug for smaller amounts. So would that bring