I’m not sure exactly what the source of your confusion is, but:
I don’t see how this follows. At the point where the confidence in PA rises above 50%, why can’t the agent be mistaken about what the theorems of PA are?
The confidence in PA as a hypothesis about what the speaker is saying is what rises above 50%. Specifically, an efficiently computable hypothesis eventually enumerating all and only the theorems of PA rises above 50%.
For example, let T be a theorem of PA that hasn’t been claimed yet. Why can’t the agent believe P(claims-T) = 0.01 and P(claims-not-T) = 0.99? It doesn’t seem like this violates any of your assumptions.
This violates the assumption of honesty that you quote, because the agent simultaneously has P(H) > 0.5 for a hypothesis H such that P(obs_n-T | H) = 1, for some (possibly very large) n, and yet also believes P(T) < 0.5. This is impossible since it must be that P(obs_n-T) > 0.5, due to P(H) > 0.5, and therefore must be that P(T) > 0.5, by honesty.
Yeah, I feel like while honesty is needed to prove the impossibility result, the problem arose with the assumption that the agent could effectively reason now about all the outputs of a recursively enumerable process (regardless of honesty). Like, the way I would phrase this point is “you can’t perfectly update on X and Carol-said-X , because you can’t have a perfect model of Carol”; this applies whether or not Carol is honest. (See also this comment.)
I agree with your first sentence, but I worry you may still be missing my point here, namely that the Bayesian notion of belief doesn’t allow us to make the distinction you are pointing to. If a hypothesis implies something, it implies it “now”; there is no “the conditional probability is 1 but that isn’t accessible to me yet”.
I also think this result has nothing to do with “you can’t have a perfect model of Carol”. Part of the point of my assumptions is that they are, individually, quite compatible with having a perfect model of Carol amongst the hypotheses.
the Bayesian notion of belief doesn’t allow us to make the distinction you are pointing to
Sure, that seems reasonable. I guess I saw this as the point of a lot of MIRI’s past work, and was expecting this to be about honesty / filtered evidence somehow.
I also think this result has nothing to do with “you can’t have a perfect model of Carol”. Part of the point of my assumptions is that they are, individually, quite compatible with having a perfect model of Carol amongst the hypotheses.
I think we mean different things by “perfect model”. What if I instead say “you can’t perfectly update on X and Carol-said-X , because you can’t know why Carol said X, because that could in the worst case require you to know everything that Carol will say in the future”?
Sure, that seems reasonable. I guess I saw this as the point of a lot of MIRI’s past work, and was expecting this to be about honesty / filtered evidence somehow.
Yeah, ok. This post as written is really less the kind of thing somebody who has followed all the MIRI thinking needs to hear and more the kind of thing one might bug an orthodox Bayesian with. I framed it in terms of filtered evidence because I came up with it by thinking about some confusion I was having about filtered evidence. And it does problematize the Bayesian treatment. But in terms of actual research progress it would be better framed as a negative result about whether Sam’s untrollable prior can be modified to have richer learning.
I think we mean different things by “perfect model”. What if [...]
I’m not sure exactly what the source of your confusion is, but:
The confidence in PA as a hypothesis about what the speaker is saying is what rises above 50%. Specifically, an efficiently computable hypothesis eventually enumerating all and only the theorems of PA rises above 50%.
This violates the assumption of honesty that you quote, because the agent simultaneously has P(H) > 0.5 for a hypothesis H such that P(obs_n-T | H) = 1, for some (possibly very large) n, and yet also believes P(T) < 0.5. This is impossible since it must be that P(obs_n-T) > 0.5, due to P(H) > 0.5, and therefore must be that P(T) > 0.5, by honesty.
Yeah, I feel like while honesty is needed to prove the impossibility result, the problem arose with the assumption that the agent could effectively reason now about all the outputs of a recursively enumerable process (regardless of honesty). Like, the way I would phrase this point is “you can’t perfectly update on X and Carol-said-X , because you can’t have a perfect model of Carol”; this applies whether or not Carol is honest. (See also this comment.)
I agree with your first sentence, but I worry you may still be missing my point here, namely that the Bayesian notion of belief doesn’t allow us to make the distinction you are pointing to. If a hypothesis implies something, it implies it “now”; there is no “the conditional probability is 1 but that isn’t accessible to me yet”.
I also think this result has nothing to do with “you can’t have a perfect model of Carol”. Part of the point of my assumptions is that they are, individually, quite compatible with having a perfect model of Carol amongst the hypotheses.
Sure, that seems reasonable. I guess I saw this as the point of a lot of MIRI’s past work, and was expecting this to be about honesty / filtered evidence somehow.
I think we mean different things by “perfect model”. What if I instead say “you can’t perfectly update on X and Carol-said-X , because you can’t know why Carol said X, because that could in the worst case require you to know everything that Carol will say in the future”?
Yeah, ok. This post as written is really less the kind of thing somebody who has followed all the MIRI thinking needs to hear and more the kind of thing one might bug an orthodox Bayesian with. I framed it in terms of filtered evidence because I came up with it by thinking about some confusion I was having about filtered evidence. And it does problematize the Bayesian treatment. But in terms of actual research progress it would be better framed as a negative result about whether Sam’s untrollable prior can be modified to have richer learning.
Yep, I agree with everything you say here.