Incoherent agents can have all manner of beliefs such as “1+1=3” and “fish are necessarily green” and “eels are not eels”.
I am not talking about incoherent agents, I am talking about agents which are coherent but not reflective. To the extent that we expect coherence to be instrumentally useful and reflection to be difficult, that’s exactly the sort of agent we should expect evolution to produce most often.
Most humans seem to have mostly-accurate beliefs, without thinking at all about whether those beliefs were systematically produced by something which produces accurate beliefs.
In interpreting the notation into a mental representation of the theorem, you need set points like “represent the theorem as a grammatical structure following these rules” and “interpret for-all claims as applying to each individual”.
It’s not at all obvious that representations and interpretations need to be implemented as set-points, or are equivalent to set points, or anything like that. That’s the claim which would be interesting to prove.
But believing one’s own beliefs to come from a source that systematically produces correct beliefs is a coherence condition. If you believe your beliefs come from source X that does not systematically produce correct beliefs, then your beliefs don’t cohere.
This can be seen in terms of Bayesianism. Let R[X] stand for “My system reports X is true”. There is no distribution P (joint over X,R[X]) such that P(X|R[X])=1 and P(X) = 0.5 and P(R[X] | X) = 1 and P(R[X] | not X) = 1.
That’s the claim which would be interesting to prove.
Here’s my attempt at a proof:
Let A stand for some reflective reasonable agent.
Axiom 1: A believes X, and A believes that A believes X.
Axiom 2: A believes that if A believes X, then there exists some epistemic system Y such that: Y contains A as an essential component, Y causes A to believe X, and Y functions well. [argument: A has internal justifications for beliefs being systematically correct. A is essential to the system because A’s beliefs are a result of the system; if not for A’s work, such beliefs would not be systematically correct]
Axiom 3: A believes that, for all epistemic systems Y that contain A as an essential component and function well, A functions well as part of Y. [argument: A is essential to Y’s functioning]
Axiom 4: For all epistemic systems Y, if A believes that Y is an epistemic system that contains A as an essential component, and also that A functions well as part of Y, then A believes that A is trying to function well as part of Y. [argument: good functioning doesn’t happen accidentally, it’s a narrow target to hit. Anyway, accidental functioning wouldn’t justify the belief; the argument has to be that the belief is systematically, not accidentally, correct.]
Axiom 5: A believes that, for all epistemic systems Y, if A is trying to function well as part of Y, then A has a set-point of functioning well as part of Y. [argument: set-point is the same as trying]
Axiom 6: For all epistemic systems Y, if A believes A has a set-point of functioning well as part of Y, then A has a set-point of functioning well as part of Y. [argument: otherwise A is incoherent; it believes itself to have a set-point it doesn’t have]
Theorem 1: A believes that there exists some epistemic system Y such that: Y contains A as an essential component, Y causes A to believe X, and Y functions well. (Follows from Axiom 1, Axiom 2)
Theorem 2: A believes that A functions well as part of Y. (Follows from Axiom 3, Theorem 1)
Theorem 3: A believes that A is trying to function well as part of Y. (Follows from Axiom 4, Theorem 2)
Theorem 4: A believes A has a set-point of functioning well as part of Y. (Follows from Axiom 5, Theorem 3)
Theorem 5: A has a set-point of functioning well as part of Y. (Follows from Axiom 6, Theorem 4)
Theorem 6: A has some set-point. (Follows from Theorem 5)
(Note, consider X = “Fermat’s last theorem universally quantifies over all triples of natural numbers”; “Fermat’s last theorem” is not meaningful to A if A lacks knowledge of X)
But believing one’s own beliefs to come from a source that systematically produces correct beliefs is a coherence condition.
This is only if you have some kind of completeness or logical omniscience kind of condition, requiring us to have beliefs about reflective statements at all. It’s entirely possible to only have beliefs over a limited class of statements—most animals don’t even have a concept of reflection, yet they have beliefs which match reality. One need not have any beliefs at all about the sources of one’s beliefs.
As for the proof, seems like the interesting part would be providing deeper foundations for axioms 4 and 5. Those are the parts which seem like they could fail.
I am not talking about incoherent agents, I am talking about agents which are coherent but not reflective. To the extent that we expect coherence to be instrumentally useful and reflection to be difficult, that’s exactly the sort of agent we should expect evolution to produce most often.
Most humans seem to have mostly-accurate beliefs, without thinking at all about whether those beliefs were systematically produced by something which produces accurate beliefs.
It’s not at all obvious that representations and interpretations need to be implemented as set-points, or are equivalent to set points, or anything like that. That’s the claim which would be interesting to prove.
But believing one’s own beliefs to come from a source that systematically produces correct beliefs is a coherence condition. If you believe your beliefs come from source X that does not systematically produce correct beliefs, then your beliefs don’t cohere.
This can be seen in terms of Bayesianism. Let R[X] stand for “My system reports X is true”. There is no distribution P (joint over X,R[X]) such that P(X|R[X])=1 and P(X) = 0.5 and P(R[X] | X) = 1 and P(R[X] | not X) = 1.
Here’s my attempt at a proof:
Let A stand for some reflective reasonable agent.
Axiom 1: A believes X, and A believes that A believes X.
Axiom 2: A believes that if A believes X, then there exists some epistemic system Y such that: Y contains A as an essential component, Y causes A to believe X, and Y functions well. [argument: A has internal justifications for beliefs being systematically correct. A is essential to the system because A’s beliefs are a result of the system; if not for A’s work, such beliefs would not be systematically correct]
Axiom 3: A believes that, for all epistemic systems Y that contain A as an essential component and function well, A functions well as part of Y. [argument: A is essential to Y’s functioning]
Axiom 4: For all epistemic systems Y, if A believes that Y is an epistemic system that contains A as an essential component, and also that A functions well as part of Y, then A believes that A is trying to function well as part of Y. [argument: good functioning doesn’t happen accidentally, it’s a narrow target to hit. Anyway, accidental functioning wouldn’t justify the belief; the argument has to be that the belief is systematically, not accidentally, correct.]
Axiom 5: A believes that, for all epistemic systems Y, if A is trying to function well as part of Y, then A has a set-point of functioning well as part of Y. [argument: set-point is the same as trying]
Axiom 6: For all epistemic systems Y, if A believes A has a set-point of functioning well as part of Y, then A has a set-point of functioning well as part of Y. [argument: otherwise A is incoherent; it believes itself to have a set-point it doesn’t have]
Theorem 1: A believes that there exists some epistemic system Y such that: Y contains A as an essential component, Y causes A to believe X, and Y functions well. (Follows from Axiom 1, Axiom 2)
Theorem 2: A believes that A functions well as part of Y. (Follows from Axiom 3, Theorem 1)
Theorem 3: A believes that A is trying to function well as part of Y. (Follows from Axiom 4, Theorem 2)
Theorem 4: A believes A has a set-point of functioning well as part of Y. (Follows from Axiom 5, Theorem 3)
Theorem 5: A has a set-point of functioning well as part of Y. (Follows from Axiom 6, Theorem 4)
Theorem 6: A has some set-point. (Follows from Theorem 5)
(Note, consider X = “Fermat’s last theorem universally quantifies over all triples of natural numbers”; “Fermat’s last theorem” is not meaningful to A if A lacks knowledge of X)
This is only if you have some kind of completeness or logical omniscience kind of condition, requiring us to have beliefs about reflective statements at all. It’s entirely possible to only have beliefs over a limited class of statements—most animals don’t even have a concept of reflection, yet they have beliefs which match reality. One need not have any beliefs at all about the sources of one’s beliefs.
As for the proof, seems like the interesting part would be providing deeper foundations for axioms 4 and 5. Those are the parts which seem like they could fail.