). The seems to be an important step along the way to eliminating the horizon problem. I recently read in Orseau and Ring’s “Space-Time Embedded Intelligence” that in another paper, “Provably Bounded-Optimal Agents” by Russell and Subramanian, they define
%20=%20u(h(\pi,%20q))) where
) generates the interaction history ao_{1:infty}. (I have yet to read this paper.)
Your counterexample of supremum chasing is really great; it breaks my model of how utility maximization is supposed to go. I’m honestly not sure whether one should chase the path of U = 0 or not. This is clouded by the fact that the probabilistic nature of things will probably push you off that path eventually.
The dilemma reminds me a lot of exploring versus exploiting. Sometimes it seems to me that the rational thing for a utility maximizer to do, almost independent of the utility function, would be to just maximize the resources it controlled, until it found some “end” or limit, and then spend all its resources creating whatever it wanted in the first place. In the framework above we’ve specified that there is no “end” time, and AIXI is dualist, so there are no worries of it getting destroyed.
There’s something else that is strange to me. If we are considering infinite interaction histories, then we’re looking at the entire binary tree at once. But this tree has uncountably infinite paths! Almost all of the (infinite) paths are incomputable sequences. This means that any computable AI couldn’t even consider traversing them. And it also seems to have interesting things to say about the utility function. Does it only need to be defined over computable sequences? What if we have utility over incomptuable sequences? These could be defined by second-order logic statements, but remain incomputable. It gives me lots of questions.
I’m honestly not sure whether one should chase the path of U = 0 or not. This is clouded by the fact that the probabilistic nature of things will probably push you off that path eventually.
Making the assumption that there is a small probability that you will deviate from your current plan on each future move, and that these probabilities add up to a near guarantee that you will eventually, has a more complicated effect on your planning than just justifying chasing the supremum.
For instance, consider this modification to the toy example I gave earlier. Y:={a,b,c}, and if the first b comes before the first c, then the resulting utility is 1 − 1/n, where n is the index of the first b (all previous elements being a), as before. But we’ll change it so that the utility of outputting an infinite stream of a is 1. If there is a c in your action sequence and it comes before the first b, then the utility you get is −1000. In this situation, supremum-chasing works just fine if you completely trust your future self: you output a every time, and get a utility of 1, the best you could possibly do. But if you think that there is a small risk that you could end up outputting c at some point, then eventually it will be worthwhile to output b, since the gain you could get from continuing to output a gets arbitrarily small compared to the loss from accidentally outputting c.
There’s something else that is strange to me. If we are considering infinite interaction histories, then we’re looking at the entire binary tree at once. But this tree has uncountably infinite paths! Almost all of the (infinite) paths are incomputable sequences. This means that any computable AI couldn’t even consider traversing them. And it also seems to have interesting things to say about the utility function. Does it only need to be defined over computable sequences? What if we have utility over incomptuable sequences? These could be defined by second-order logic statements, but remain incomputable. It gives me lots of questions.
I don’t really have answers to these questions. One thing you could do is replace the set of all policies (P) with the set of all computable policies, so that the agent would never output an uncomputable action sequence [Edit: oops, not true. You could consider only computable policies, but then end up at an uncomputable policy anyway by chasing the supremum].
Whether the agent outputs an uncomputable sequence of actions isn’t really a concern, but the environment frequently does (some l.s.c. environments output uniformly random percepts).
Yeah, I was intentionally vague with “the probabilistic nature of things”. I am also thinking about how any AI will have logical uncertainty, uncertainty about the precision of its observations, et cetera, so that as it considers further points in the future, its distribution becomes flatter. And having a non-dualist framework would introduce uncertainty about the agent’s self, its utility function, its memory, …
This is great!
I really like your use of
Your counterexample of supremum chasing is really great; it breaks my model of how utility maximization is supposed to go. I’m honestly not sure whether one should chase the path of U = 0 or not. This is clouded by the fact that the probabilistic nature of things will probably push you off that path eventually.
The dilemma reminds me a lot of exploring versus exploiting. Sometimes it seems to me that the rational thing for a utility maximizer to do, almost independent of the utility function, would be to just maximize the resources it controlled, until it found some “end” or limit, and then spend all its resources creating whatever it wanted in the first place. In the framework above we’ve specified that there is no “end” time, and AIXI is dualist, so there are no worries of it getting destroyed.
There’s something else that is strange to me. If we are considering infinite interaction histories, then we’re looking at the entire binary tree at once. But this tree has uncountably infinite paths! Almost all of the (infinite) paths are incomputable sequences. This means that any computable AI couldn’t even consider traversing them. And it also seems to have interesting things to say about the utility function. Does it only need to be defined over computable sequences? What if we have utility over incomptuable sequences? These could be defined by second-order logic statements, but remain incomputable. It gives me lots of questions.
The draft I recently sent you answers some of these questions. The expected utility should be an appropriately defined integral not a sum.
Making the assumption that there is a small probability that you will deviate from your current plan on each future move, and that these probabilities add up to a near guarantee that you will eventually, has a more complicated effect on your planning than just justifying chasing the supremum.
For instance, consider this modification to the toy example I gave earlier. Y:={a,b,c}, and if the first b comes before the first c, then the resulting utility is 1 − 1/n, where n is the index of the first b (all previous elements being a), as before. But we’ll change it so that the utility of outputting an infinite stream of a is 1. If there is a c in your action sequence and it comes before the first b, then the utility you get is −1000. In this situation, supremum-chasing works just fine if you completely trust your future self: you output a every time, and get a utility of 1, the best you could possibly do. But if you think that there is a small risk that you could end up outputting c at some point, then eventually it will be worthwhile to output b, since the gain you could get from continuing to output a gets arbitrarily small compared to the loss from accidentally outputting c.
I don’t really have answers to these questions. One thing you could do is replace the set of all policies (P) with the set of all computable policies, so that the agent would never output an uncomputable action sequence [Edit: oops, not true. You could consider only computable policies, but then end up at an uncomputable policy anyway by chasing the supremum].
Whether the agent outputs an uncomputable sequence of actions isn’t really a concern, but the environment frequently does (some l.s.c. environments output uniformly random percepts).
Yeah, I was intentionally vague with “the probabilistic nature of things”. I am also thinking about how any AI will have logical uncertainty, uncertainty about the precision of its observations, et cetera, so that as it considers further points in the future, its distribution becomes flatter. And having a non-dualist framework would introduce uncertainty about the agent’s self, its utility function, its memory, …