I am glad that a term like “bounded-rational” exists. If it’s been discussed someplace very thoroughly then I likely don’t have very much to add. What are some proposals for modeling bounded Bayesianism?
I think what I’m saying is consistent with your bullet points, but I would go further. I’ll focus on one point: I do not think it’s possible for a bounded agent to be epistemically pure, even having sacrificed most or all of its instrumentally rational capability. Epistemic impurity is built right into math and logic.
Let me make the following assumption about our bounded rational agent: given any assertion A, it has the capability of computing its prior P(A) in time that is polynomial in the length of A. That is, it is not strictly agnostic about anything. Since there exist assertions A which are logical consequences of some axioms, but whose shortest proof is super-polynomial (in fact it gets much worse) in the length of A, it seem very unlikely that we will have P(A) > P(Axioms) for all provable assertions A.
(I think you could make this into a rigorous mathematical statement, but I am not claiming to have proved it—I don’t see how to rule out the possibility that P always computes P(A) > P(Axioms) (and quickly!) just by luck. Such a P would be very valuable.)
I believe you are correct. A bounded rational agent that is not strictly agnostic about anything will produce outputs that contain logical inconsistencies.
For a superintelligence to avoid such inconsistencies it would have to violate the ‘not strictly agnostic about anything’ assumption either explicitly or practically. By ‘practically’ I mean it could refuse to return output until such time as it has proven the logical correctness of a given A. It may burn up the neg-entropy of its future light cone before it returns but hey, at least it was never wrong. A bounded rational agent in denial about its ‘bounds’.
I had in mind to rule out your “practical agnosticism” with the polynomial time condition. Note that we’re talking about the zeroth thing that an intelligence is supposed to do, not “learning” or “deciding” but just “telling us (or itself) what it believes.” In toy problems about balls in urns (and maybe, problematically, more general examples) this is often implicitly assumed to be an instantaneous process.
If we’re going to allow explicit agnosticism, we’re going to have to rethink some things. If P(A) = refuse to answer, what are P(B|A) and P(A|B)? How are we supposed to update?
I had in mind to rule out your “practical agnosticism” with the polynomial time condition.
That is a reasonable assumption to make. We just need to explicitly assert that the intelligence is willing and able to return P(A) for any sane length A that matches the polynomial time condition. (And so explicitly rule out intelligences that just compute perfect answers and to hell with polynomial time limits and pesky things like physical possibility.)
If we’re going to allow explicit agnosticism, we’re going to have to rethink some things. If P(A) = refuse to answer, what are P(B|A) and P(A|B)? How are we supposed to update?
I don’t know and the intelligence doesn’t care. It just isn’t going to give you wrong answers. I think it is reasonable for us to just exclude such intelligences because they are practically useless. I’ll include the same caveat that you mentioned earlier—maybe there is some algorithm that never violates logical consistency conditions somehow. That algorithm would be an extremely valuable discovery but one I suspect could be proven impossible. The maths for making such a proof is beyond me.
I am glad that a term like “bounded-rational” exists. If it’s been discussed someplace very thoroughly then I likely don’t have very much to add. What are some proposals for modeling bounded Bayesianism?
I think what I’m saying is consistent with your bullet points, but I would go further. I’ll focus on one point: I do not think it’s possible for a bounded agent to be epistemically pure, even having sacrificed most or all of its instrumentally rational capability. Epistemic impurity is built right into math and logic.
Let me make the following assumption about our bounded rational agent: given any assertion A, it has the capability of computing its prior P(A) in time that is polynomial in the length of A. That is, it is not strictly agnostic about anything. Since there exist assertions A which are logical consequences of some axioms, but whose shortest proof is super-polynomial (in fact it gets much worse) in the length of A, it seem very unlikely that we will have P(A) > P(Axioms) for all provable assertions A.
(I think you could make this into a rigorous mathematical statement, but I am not claiming to have proved it—I don’t see how to rule out the possibility that P always computes P(A) > P(Axioms) (and quickly!) just by luck. Such a P would be very valuable.)
I believe you are correct. A bounded rational agent that is not strictly agnostic about anything will produce outputs that contain logical inconsistencies.
For a superintelligence to avoid such inconsistencies it would have to violate the ‘not strictly agnostic about anything’ assumption either explicitly or practically. By ‘practically’ I mean it could refuse to return output until such time as it has proven the logical correctness of a given A. It may burn up the neg-entropy of its future light cone before it returns but hey, at least it was never wrong. A bounded rational agent in denial about its ‘bounds’.
I see how to prove my claim now, more later.
I had in mind to rule out your “practical agnosticism” with the polynomial time condition. Note that we’re talking about the zeroth thing that an intelligence is supposed to do, not “learning” or “deciding” but just “telling us (or itself) what it believes.” In toy problems about balls in urns (and maybe, problematically, more general examples) this is often implicitly assumed to be an instantaneous process.
If we’re going to allow explicit agnosticism, we’re going to have to rethink some things. If P(A) = refuse to answer, what are P(B|A) and P(A|B)? How are we supposed to update?
That is a reasonable assumption to make. We just need to explicitly assert that the intelligence is willing and able to return P(A) for any sane length A that matches the polynomial time condition. (And so explicitly rule out intelligences that just compute perfect answers and to hell with polynomial time limits and pesky things like physical possibility.)
I don’t know and the intelligence doesn’t care. It just isn’t going to give you wrong answers. I think it is reasonable for us to just exclude such intelligences because they are practically useless. I’ll include the same caveat that you mentioned earlier—maybe there is some algorithm that never violates logical consistency conditions somehow. That algorithm would be an extremely valuable discovery but one I suspect could be proven impossible. The maths for making such a proof is beyond me.