That said, I’m not sure what use free-form speculating about such bizarre and underspecified scenarios really is, though I’ll admit it’s kind of fun.
It’s kind of fun. Isn’t that reason enough?
Looking at the original question—i.e. how to handle very large utilities with very small probability—I find that I have a mental safety net there. The safety net says that the situation is a lie. It does not matter how much utility is claimed, because anyone can state any arbitrarily large number, and a number has been chosen (in this case, by the Matrix Lord) in a specific attempt to overwhelm my utility function. The small probability is chosen (a) because I would not believe a larger probability and (b) so that I have no recourse when it fails to happen.
I am reluctant to fiddle with my mental safety nets because, well, they’re safety nets—they’re there for a reason. And in this case, the reason is that such a fantastically unlikely event is unlikely enough that it’s not likely to happen ever, to anyone. Not even once in the whole history of the universe. If I (out of all the hundreds of billions of people in all of history) do ever run across such a situation, then it’s so incredibly overwhelmingly more likely that I am being deceived that I’m far more likely to gain by immediately jumping to the conclusion of ‘deceit’ than by assuming that there’s any chance of this being true.
It’s kind of fun. Isn’t that reason enough?
Looking at the original question—i.e. how to handle very large utilities with very small probability—I find that I have a mental safety net there. The safety net says that the situation is a lie. It does not matter how much utility is claimed, because anyone can state any arbitrarily large number, and a number has been chosen (in this case, by the Matrix Lord) in a specific attempt to overwhelm my utility function. The small probability is chosen (a) because I would not believe a larger probability and (b) so that I have no recourse when it fails to happen.
I am reluctant to fiddle with my mental safety nets because, well, they’re safety nets—they’re there for a reason. And in this case, the reason is that such a fantastically unlikely event is unlikely enough that it’s not likely to happen ever, to anyone. Not even once in the whole history of the universe. If I (out of all the hundreds of billions of people in all of history) do ever run across such a situation, then it’s so incredibly overwhelmingly more likely that I am being deceived that I’m far more likely to gain by immediately jumping to the conclusion of ‘deceit’ than by assuming that there’s any chance of this being true.
(nods) Sure. My reply here applies here as well.