This comment thread has grown too large :). I have a thought that seems to me to be the right way to resolve this problem. On the one hand, the thought is obvious, so it probably has already been played out in this comment thread, where it presumably failed to convince everyone. On the other hand, the thread is too large for me to digest in the time that I can reasonably give it. So I’m hoping that someone more familiar with the conversation here will tell me where I can find the sub-thread that addresses my point. (I tried some obvious word-searches, and nothing came up.)
Anyway, here is my point. I can see that the hypothesis that 3^^^^3 people are being tortured might be simple enough so that the Solomonoff prior is high enough so that the AI would give in to the mugger, if the AI were using an un-updated Solomonoff prior. But the AI is allowed to update, right? And, from what the AI knows about humans, it can see that the low complexity of 3^^^^3 also makes it more probable that a “philosopher out for a fast buck” would choose that number.
So, the simplicity of 3^^^^3 contributes to both the hypothesis of a real torturer and the hypothesis of the liar.
And if, after taking all this into account, the AI still computes a high expected utility for giving in to the mugger, well, then I guess that that is really what it ought to do (assuming that it shares my utility function). But is there any reason to think that this is likely? Does it follow just from Eliezer’s observation that “the utility of a Turing machine can grow much faster than its prior probability shrinks”? After all, it’s the updated probability that really matters, isn’t it?
This comment thread has grown too large :). I have a thought that seems to me to be the right way to resolve this problem. On the one hand, the thought is obvious, so it probably has already been played out in this comment thread, where it presumably failed to convince everyone. On the other hand, the thread is too large for me to digest in the time that I can reasonably give it. So I’m hoping that someone more familiar with the conversation here will tell me where I can find the sub-thread that addresses my point. (I tried some obvious word-searches, and nothing came up.)
Anyway, here is my point. I can see that the hypothesis that 3^^^^3 people are being tortured might be simple enough so that the Solomonoff prior is high enough so that the AI would give in to the mugger, if the AI were using an un-updated Solomonoff prior. But the AI is allowed to update, right? And, from what the AI knows about humans, it can see that the low complexity of 3^^^^3 also makes it more probable that a “philosopher out for a fast buck” would choose that number.
So, the simplicity of 3^^^^3 contributes to both the hypothesis of a real torturer and the hypothesis of the liar.
And if, after taking all this into account, the AI still computes a high expected utility for giving in to the mugger, well, then I guess that that is really what it ought to do (assuming that it shares my utility function). But is there any reason to think that this is likely? Does it follow just from Eliezer’s observation that “the utility of a Turing machine can grow much faster than its prior probability shrinks”? After all, it’s the updated probability that really matters, isn’t it?
That assumption is wrong, I argue.
I missed your post when it first came out. I’ve just commented on it.