Following is a generalization to take into account logical uncertainty (inability to prove difficult statements, which can lead in particular to inability to prove any statements of the necessary kind).
Instead of proving statements of the form “agent()==A implies world()==U”, let’s prove statements of the form “(X and agent()==A) implies world()==U”, where X are unsubstantiated assumptions. There are obviously provable statements of the new form. Now, instead of looking for a statement with highest U, let’s for each A look at the sets of pairs (X,U) for proved statements. We can now use the inverse of length of X to weight the corresponding utilities and estimate the “expected utility” of performing action A by summing over all explored X, even if we are unable to complete a single proof of the statements of the original form. For the same reason as in the post, this “expected utility” calculation is valid for the action that we choose.
(This looks like a potentially dangerous algorithm.)
Edit: As cousin_it points out, it doesn’t quite work as intended, because X could just be “false”, which would enable arbitrarily high U and collapse the “expected utility” calculation.
If you sum over all explored X (and implicitly treat unexplored X as if they are zero probability) then you can be arbitrarily far from optimal Bayesian behavior.
See Fitelson’s “Bayesians sometimes cannot ignore even very implausible theories.”
What if instead of assuming your Xs, you got them out of the program?
To simplify, imagine a universe with only one fact. A function fact() returns 1 if that fact is true, and 0 if its not.
Now, agent can prove statements of the form “fact()==1 and agent()==A implies world()==U”. This avoids the problem of plugging in false, agent doesn’t start from assuming X as a statement of its own from which it can derive things, it looks at all the possible outputs of fact(), then sees how that combines with its action to produce utility.
Trivially, if agent has access to the code of fact(), then it can just figure out what that result would be, and it would know what actions correspond to what utilities.
Otherwise, agent() could use a prior distribution or magically infer some probability distribution of if fact is true or not, then choose its action based on normal expected utility calculations.
Note: The above departs from the structure of the world() function given here in that it assumes some way of interacting with world (or its source code) to find out fact() without actually getting the code of fact. Perhaps world is called multiple times, and agent() can observe the utility it accumulates and use that (combined with its “agent()==A and fact()==1 implies world() == U” proofs) to figure out if fact is true or not?
Interestingly, while this allows agent to apply “could” to the world (“It could rain tomorrow”), it doesn’t allow the agent to apply “could” to things that don’t effect world’s utility calculation (“People could have nonmaterial souls that can be removed without influencing their behavior in any way”).
Following is a generalization to take into account logical uncertainty (inability to prove difficult statements, which can lead in particular to inability to prove any statements of the necessary kind).
Instead of proving statements of the form “agent()==A implies world()==U”, let’s prove statements of the form “(X and agent()==A) implies world()==U”, where X are unsubstantiated assumptions. There are obviously provable statements of the new form. Now, instead of looking for a statement with highest U, let’s for each A look at the sets of pairs (X,U) for proved statements. We can now use the inverse of length of X to weight the corresponding utilities and estimate the “expected utility” of performing action A by summing over all explored X, even if we are unable to complete a single proof of the statements of the original form. For the same reason as in the post, this “expected utility” calculation is valid for the action that we choose.
(This looks like a potentially dangerous algorithm.)
Edit: As cousin_it points out, it doesn’t quite work as intended, because X could just be “false”, which would enable arbitrarily high U and collapse the “expected utility” calculation.
Ouch. X could simply say “false” and the whole thing explodes in your face.
There are logics (relevant or paraconsistent) logics which do not have the classical “ex falso quodlibet”.
True, didn’t notice that. I still feel there must be a workaround.
If you sum over all explored X (and implicitly treat unexplored X as if they are zero probability) then you can be arbitrarily far from optimal Bayesian behavior.
See Fitelson’s “Bayesians sometimes cannot ignore even very implausible theories.”
http://fitelson.org/hiti.pdf
Thanks for that link. Good stuff.
What if instead of assuming your Xs, you got them out of the program?
To simplify, imagine a universe with only one fact. A function fact() returns 1 if that fact is true, and 0 if its not.
Now, agent can prove statements of the form “fact()==1 and agent()==A implies world()==U”. This avoids the problem of plugging in false, agent doesn’t start from assuming X as a statement of its own from which it can derive things, it looks at all the possible outputs of fact(), then sees how that combines with its action to produce utility.
Trivially, if agent has access to the code of fact(), then it can just figure out what that result would be, and it would know what actions correspond to what utilities.
Otherwise, agent() could use a prior distribution or magically infer some probability distribution of if fact is true or not, then choose its action based on normal expected utility calculations.
Note: The above departs from the structure of the world() function given here in that it assumes some way of interacting with world (or its source code) to find out fact() without actually getting the code of fact. Perhaps world is called multiple times, and agent() can observe the utility it accumulates and use that (combined with its “agent()==A and fact()==1 implies world() == U” proofs) to figure out if fact is true or not?
Interestingly, while this allows agent to apply “could” to the world (“It could rain tomorrow”), it doesn’t allow the agent to apply “could” to things that don’t effect world’s utility calculation (“People could have nonmaterial souls that can be removed without influencing their behavior in any way”).