I think the problem might lie in the almost laughable disparity between the price and the possible risk. A human mind is not capable of instinctively providing a reason why it would be worth killing 3^^^^3 people—or even, I think, a million people—as punishment for not getting $5. A mind who would value $5 as much or more than the lives of 3^^^^3 people is utterly alien to us, and so we leap to the much more likely assumption that the guy is crazy.
Is this a bias? I’d call it a heuristic. It calls to my mind the discussion in Neal Stephenson’s Anathem about pink nerve-gas-farting dragons. (Mandatory warning: fictional example.) The crux of it is, our minds only bother to anticipate situations that we can conceive of as logical. Therefore, the manifest illogicality of the mugging (why is 3^^^^3 lives worth $5; if you’re a Matrix Lord why can’t you just generate $5 or better yet, modify my mind so that I’m inclined to give you $5, etc.) causes us to anti-anticipate its truth. Otherwise, what’s to stop you from imagining, as stated by Tom_McCabe2 (and mitchell_porter2, &c.), that typing the string “QWERTYUIOP” leads to, for example, 3^^^^3 deaths? If you imagine it, and conceive of it as a logically possible outcome,
then regardless of its improbability, by your argument (as I see it), a “mind that worked strictly by Solomonoff induction” should cease to type that string of letters ever again. By induction, such a mind could cause itself to cease to take any action, which would lead to… well, if the AI had access to itself, likely self-deletion.
That’s my top-of-the-head theory. It doesn’t really answer the question at hand, but maybe I’m on the right track...?
I think the problem might lie in the almost laughable disparity between the price and the possible risk. A human mind is not capable of instinctively providing a reason why it would be worth killing 3^^^^3 people—or even, I think, a million people—as punishment for not getting $5. A mind who would value $5 as much or more than the lives of 3^^^^3 people is utterly alien to us, and so we leap to the much more likely assumption that the guy is crazy.
Is this a bias? I’d call it a heuristic. It calls to my mind the discussion in Neal Stephenson’s Anathem about pink nerve-gas-farting dragons. (Mandatory warning: fictional example.) The crux of it is, our minds only bother to anticipate situations that we can conceive of as logical. Therefore, the manifest illogicality of the mugging (why is 3^^^^3 lives worth $5; if you’re a Matrix Lord why can’t you just generate $5 or better yet, modify my mind so that I’m inclined to give you $5, etc.) causes us to anti-anticipate its truth. Otherwise, what’s to stop you from imagining, as stated by Tom_McCabe2 (and mitchell_porter2, &c.), that typing the string “QWERTYUIOP” leads to, for example, 3^^^^3 deaths? If you imagine it, and conceive of it as a logically possible outcome, then regardless of its improbability, by your argument (as I see it), a “mind that worked strictly by Solomonoff induction” should cease to type that string of letters ever again. By induction, such a mind could cause itself to cease to take any action, which would lead to… well, if the AI had access to itself, likely self-deletion.
That’s my top-of-the-head theory. It doesn’t really answer the question at hand, but maybe I’m on the right track...?