If there’s a subset of option 2 that still lets people be perfect reasoners, I’d love to hear it—that might be the most interesting part of the puzzle
Make them forget about some piece of evidence they already updated on! Let’s say that evidence A moves B in some direction, and that P(B|A) has already been computed. If you forget A, by encountering A again you would get P(B|A,A) <> P(B|A), while still executing the perfect reasoner’s algorithm.
People can counteract that trick and other similar tricks by constantly regenerating their beliefs from their original prior and remembered evidence. Can you make a more watertight model?
I think we can combine your [cousin_it’s] suggestion with MrMind’s for an Option 2 scenario.
Suppose Bob finds that he has a stored belief in Bright with an apparent memory of having based it on evidence A, but no memory of what evidence A was. That does constitute some small evidence in favor of Bright existing.
But if Bob then goes out in search of evidence about whether Bright exists, and finds some evidence A in favor, he is unable to know whether it’s the same evidence as before that he had forgotten, or if it’s different evidence. Another way of saying that is that Bob can’t tell whether or not A and A are independent. I suppose the ideal reasoner’s response would be to assign a probability density distribution over a range from full independence to full dependence and proceed with any belief updates taking that distribution into account.
The distribution should be formed by consideration of how Bob got the evidence. If Bob found his new evidence A in some easily repeatable way, like hearing it from Bright apologists, then Bob would probably think dependence on A is much more likely than independence, and so he would take into account mostly just A and not A. But if Bob got A by some means that he probably wouldn’t have had access to in the past, like an experiment requiring brand new technology to perform, then he would probably think independence was more likely, and so he would take into account A and A mostly separately.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior (at least from the subjective view of an external observer) by ridding them (possibly temporarily) of key facts.
And then there is the question of whether they may notice this as some inferences are more likely to be detected when you already have some other facts.
I’d guess that you’d quickly notice if you should suddenly have forgotten that you were repeatedly told that Bright exists.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior
Surely I can construct such a model. But whether this is generally the case depends too much on the details of the implementation to give a complete answers.
whether they may notice this as some inferences are more likely to be detected when you already have some other facts.
… especially logical inferences: logical deductions of true facts are true, even if you don’t know/remember them. But then again, that depends too much on the implementation of the agent to have a general answer, in this case also its computational power would matter.
Make them forget about some piece of evidence they already updated on!
Let’s say that evidence A moves B in some direction, and that P(B|A) has already been computed. If you forget A, by encountering A again you would get P(B|A,A) <> P(B|A), while still executing the perfect reasoner’s algorithm.
People can counteract that trick and other similar tricks by constantly regenerating their beliefs from their original prior and remembered evidence. Can you make a more watertight model?
I think we can combine your [cousin_it’s] suggestion with MrMind’s for an Option 2 scenario.
Suppose Bob finds that he has a stored belief in Bright with an apparent memory of having based it on evidence A, but no memory of what evidence A was. That does constitute some small evidence in favor of Bright existing.
But if Bob then goes out in search of evidence about whether Bright exists, and finds some evidence A in favor, he is unable to know whether it’s the same evidence as before that he had forgotten, or if it’s different evidence. Another way of saying that is that Bob can’t tell whether or not A and A are independent. I suppose the ideal reasoner’s response would be to assign a probability density distribution over a range from full independence to full dependence and proceed with any belief updates taking that distribution into account.
The distribution should be formed by consideration of how Bob got the evidence. If Bob found his new evidence A in some easily repeatable way, like hearing it from Bright apologists, then Bob would probably think dependence on A is much more likely than independence, and so he would take into account mostly just A and not A. But if Bob got A by some means that he probably wouldn’t have had access to in the past, like an experiment requiring brand new technology to perform, then he would probably think independence was more likely, and so he would take into account A and A mostly separately.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior (at least from the subjective view of an external observer) by ridding them (possibly temporarily) of key facts.
And then there is the question of whether they may notice this as some inferences are more likely to be detected when you already have some other facts. I’d guess that you’d quickly notice if you should suddenly have forgotten that you were repeatedly told that Bright exists.
Surely I can construct such a model. But whether this is generally the case depends too much on the details of the implementation to give a complete answers.
… especially logical inferences: logical deductions of true facts are true, even if you don’t know/remember them. But then again, that depends too much on the implementation of the agent to have a general answer, in this case also its computational power would matter.