One way to model someone’s beliefs, at a given frozen moment of time, is as a real-valued function P on the set of all assertions. In an ideal situation, P will be subject to a lot of consistency conditions, for instance if A is a logical consequence of B, then P(A) is not smaller than P(B). This ideal P is very smart: if such a P has P(math axioms) very close to 1, then it will have P(math theorems) very close to 1 as well.
Clearly, even a Bayesian superintelligence is not going to maintain an infinitely large database of values of P, that it updates from instant to instant. Rather, it will have something like a computer program that takes as inputs assertions A, spends some time thinking, and outputs numbers P(A). I think we cannot expect the computed numbers P(A) to have the consistency property (B implies A means P(A) not smaller than P(B)). For instance it should be possible for a superintelligence to answer a math question (I don’t know, Goldbach’s conjecture) with “very likely true” and have Goldbach’s conjecture turn out false.
(Since “A” is a logical consequence of “A and B”, I guess I am accusing superintelligences of committing a souped-up form of the conjunction fallacy.)
The fact that a prior in practice won’t be a set of cached numbers but instead a computer program, subject to all the attendant resource constraints, seems important to me, but I’m open to the possibility that it’s a red herring. Am I making some kind of classic or easily addressed error?
I think we cannot expect the computed numbers P(A) to have the consistency property (B implies A means P(A) not smaller than P(B)).
Clarify for me what you are saying here. Why would a bounded-rational superintelligence maintain a logically inconsistent belief system? Are you making the following observation?
Even a superintelligence is not logically omniscient
There are inevitably going to be complicated mathematical properties of the superintelligence’s map that are not worth spending processing time on. This is ensured by the limits of physics itself.
Outside the ‘bounds’ of the superintelligence’s logical searching there will be logical properties for which the superintelligence’s beliefs are not consistent.
A bayesian superintelligence is not logically omniscient so some of it’s beliefs will be logically inconsistent.
My impression is that the above holds true unless the superintelligence in question cripples itself, sacrificing most of it’s instrumentally rational capability for epistemic purity.
I am glad that a term like “bounded-rational” exists. If it’s been discussed someplace very thoroughly then I likely don’t have very much to add. What are some proposals for modeling bounded Bayesianism?
I think what I’m saying is consistent with your bullet points, but I would go further. I’ll focus on one point: I do not think it’s possible for a bounded agent to be epistemically pure, even having sacrificed most or all of its instrumentally rational capability. Epistemic impurity is built right into math and logic.
Let me make the following assumption about our bounded rational agent: given any assertion A, it has the capability of computing its prior P(A) in time that is polynomial in the length of A. That is, it is not strictly agnostic about anything. Since there exist assertions A which are logical consequences of some axioms, but whose shortest proof is super-polynomial (in fact it gets much worse) in the length of A, it seem very unlikely that we will have P(A) > P(Axioms) for all provable assertions A.
(I think you could make this into a rigorous mathematical statement, but I am not claiming to have proved it—I don’t see how to rule out the possibility that P always computes P(A) > P(Axioms) (and quickly!) just by luck. Such a P would be very valuable.)
I believe you are correct. A bounded rational agent that is not strictly agnostic about anything will produce outputs that contain logical inconsistencies.
For a superintelligence to avoid such inconsistencies it would have to violate the ‘not strictly agnostic about anything’ assumption either explicitly or practically. By ‘practically’ I mean it could refuse to return output until such time as it has proven the logical correctness of a given A. It may burn up the neg-entropy of its future light cone before it returns but hey, at least it was never wrong. A bounded rational agent in denial about its ‘bounds’.
I had in mind to rule out your “practical agnosticism” with the polynomial time condition. Note that we’re talking about the zeroth thing that an intelligence is supposed to do, not “learning” or “deciding” but just “telling us (or itself) what it believes.” In toy problems about balls in urns (and maybe, problematically, more general examples) this is often implicitly assumed to be an instantaneous process.
If we’re going to allow explicit agnosticism, we’re going to have to rethink some things. If P(A) = refuse to answer, what are P(B|A) and P(A|B)? How are we supposed to update?
I had in mind to rule out your “practical agnosticism” with the polynomial time condition.
That is a reasonable assumption to make. We just need to explicitly assert that the intelligence is willing and able to return P(A) for any sane length A that matches the polynomial time condition. (And so explicitly rule out intelligences that just compute perfect answers and to hell with polynomial time limits and pesky things like physical possibility.)
If we’re going to allow explicit agnosticism, we’re going to have to rethink some things. If P(A) = refuse to answer, what are P(B|A) and P(A|B)? How are we supposed to update?
I don’t know and the intelligence doesn’t care. It just isn’t going to give you wrong answers. I think it is reasonable for us to just exclude such intelligences because they are practically useless. I’ll include the same caveat that you mentioned earlier—maybe there is some algorithm that never violates logical consistency conditions somehow. That algorithm would be an extremely valuable discovery but one I suspect could be proven impossible. The maths for making such a proof is beyond me.
It is the difference between what you describe in the first paragraph and what an actually possible agent is limited to. The kind of thing that allows a superintelligence that has no flaws to still be wrong about something like Goldbach’s conjecture. Always thinking perfectly but not able to think infinitely fast and about everything.
and I don’t know what “instrumentally rational” means.
Achieving your goals in the best possible way. In contrast to ‘epistemic rationality’ is having the best beliefs possible. The concepts are related but distinct.
One way to model someone’s beliefs, at a given frozen moment of time, is as a real-valued function P on the set of all assertions. In an ideal situation, P will be subject to a lot of consistency conditions, for instance if A is a logical consequence of B, then P(A) is not smaller than P(B). This ideal P is very smart: if such a P has P(math axioms) very close to 1, then it will have P(math theorems) very close to 1 as well.
Clearly, even a Bayesian superintelligence is not going to maintain an infinitely large database of values of P, that it updates from instant to instant. Rather, it will have something like a computer program that takes as inputs assertions A, spends some time thinking, and outputs numbers P(A). I think we cannot expect the computed numbers P(A) to have the consistency property (B implies A means P(A) not smaller than P(B)). For instance it should be possible for a superintelligence to answer a math question (I don’t know, Goldbach’s conjecture) with “very likely true” and have Goldbach’s conjecture turn out false.
(Since “A” is a logical consequence of “A and B”, I guess I am accusing superintelligences of committing a souped-up form of the conjunction fallacy.)
The fact that a prior in practice won’t be a set of cached numbers but instead a computer program, subject to all the attendant resource constraints, seems important to me, but I’m open to the possibility that it’s a red herring. Am I making some kind of classic or easily addressed error?
Clarify for me what you are saying here. Why would a bounded-rational superintelligence maintain a logically inconsistent belief system? Are you making the following observation?
Even a superintelligence is not logically omniscient
There are inevitably going to be complicated mathematical properties of the superintelligence’s map that are not worth spending processing time on. This is ensured by the limits of physics itself.
Outside the ‘bounds’ of the superintelligence’s logical searching there will be logical properties for which the superintelligence’s beliefs are not consistent.
A bayesian superintelligence is not logically omniscient so some of it’s beliefs will be logically inconsistent.
My impression is that the above holds true unless the superintelligence in question cripples itself, sacrificing most of it’s instrumentally rational capability for epistemic purity.
I am glad that a term like “bounded-rational” exists. If it’s been discussed someplace very thoroughly then I likely don’t have very much to add. What are some proposals for modeling bounded Bayesianism?
I think what I’m saying is consistent with your bullet points, but I would go further. I’ll focus on one point: I do not think it’s possible for a bounded agent to be epistemically pure, even having sacrificed most or all of its instrumentally rational capability. Epistemic impurity is built right into math and logic.
Let me make the following assumption about our bounded rational agent: given any assertion A, it has the capability of computing its prior P(A) in time that is polynomial in the length of A. That is, it is not strictly agnostic about anything. Since there exist assertions A which are logical consequences of some axioms, but whose shortest proof is super-polynomial (in fact it gets much worse) in the length of A, it seem very unlikely that we will have P(A) > P(Axioms) for all provable assertions A.
(I think you could make this into a rigorous mathematical statement, but I am not claiming to have proved it—I don’t see how to rule out the possibility that P always computes P(A) > P(Axioms) (and quickly!) just by luck. Such a P would be very valuable.)
I believe you are correct. A bounded rational agent that is not strictly agnostic about anything will produce outputs that contain logical inconsistencies.
For a superintelligence to avoid such inconsistencies it would have to violate the ‘not strictly agnostic about anything’ assumption either explicitly or practically. By ‘practically’ I mean it could refuse to return output until such time as it has proven the logical correctness of a given A. It may burn up the neg-entropy of its future light cone before it returns but hey, at least it was never wrong. A bounded rational agent in denial about its ‘bounds’.
I see how to prove my claim now, more later.
I had in mind to rule out your “practical agnosticism” with the polynomial time condition. Note that we’re talking about the zeroth thing that an intelligence is supposed to do, not “learning” or “deciding” but just “telling us (or itself) what it believes.” In toy problems about balls in urns (and maybe, problematically, more general examples) this is often implicitly assumed to be an instantaneous process.
If we’re going to allow explicit agnosticism, we’re going to have to rethink some things. If P(A) = refuse to answer, what are P(B|A) and P(A|B)? How are we supposed to update?
That is a reasonable assumption to make. We just need to explicitly assert that the intelligence is willing and able to return P(A) for any sane length A that matches the polynomial time condition. (And so explicitly rule out intelligences that just compute perfect answers and to hell with polynomial time limits and pesky things like physical possibility.)
I don’t know and the intelligence doesn’t care. It just isn’t going to give you wrong answers. I think it is reasonable for us to just exclude such intelligences because they are practically useless. I’ll include the same caveat that you mentioned earlier—maybe there is some algorithm that never violates logical consistency conditions somehow. That algorithm would be an extremely valuable discovery but one I suspect could be proven impossible. The maths for making such a proof is beyond me.
I don’t know what “bounded-rational” means, and I don’t know what “instrumentally rational” means. More soon, probably today.
It is the difference between what you describe in the first paragraph and what an actually possible agent is limited to. The kind of thing that allows a superintelligence that has no flaws to still be wrong about something like Goldbach’s conjecture. Always thinking perfectly but not able to think infinitely fast and about everything.
Achieving your goals in the best possible way. In contrast to ‘epistemic rationality’ is having the best beliefs possible. The concepts are related but distinct.