Attempting to approach goal directedness behaviorally is, I expect, going to run into the same problems as trying to infer policy from behaviors only: you can’t do it unless you make some normative assumption. This is exactly analogous to the Armstrong’s No Free Lunch Theorem for value learning and, to turn it around the other way, we can similarly assign any goal whatsoever to a system based solely on its behavior unless we make some sufficiently strong normative assumption about it.
That’s a very good point. I actually think we can avoid this problem, due to a couple of things:
As I mentioned in another comment, what I mean by behaviorally is not simply looking at the behavior, it also includes taking the intentional stance towards the system. And therefore making rather strong normative assumptions about it.
If we use focus, then not all systems are maximally focused towards all goals. Where I think the problem creeps back in is in the fact that many goals (like the one containing all states, which means intuitively that the goal is to reach any state) will be maximally focused for many if not all systems. My attempt at an answer is the triviality measure of the goal, as a counterweight. But it’s still possible in theory to have two goals of equivalent triviality and equivalent focus; in that case I don’t really know yet how to “choose”.
Attempting to approach goal directedness behaviorally is, I expect, going to run into the same problems as trying to infer policy from behaviors only: you can’t do it unless you make some normative assumption. This is exactly analogous to the Armstrong’s No Free Lunch Theorem for value learning and, to turn it around the other way, we can similarly assign any goal whatsoever to a system based solely on its behavior unless we make some sufficiently strong normative assumption about it.
That’s a very good point. I actually think we can avoid this problem, due to a couple of things:
As I mentioned in another comment, what I mean by behaviorally is not simply looking at the behavior, it also includes taking the intentional stance towards the system. And therefore making rather strong normative assumptions about it.
If we use focus, then not all systems are maximally focused towards all goals. Where I think the problem creeps back in is in the fact that many goals (like the one containing all states, which means intuitively that the goal is to reach any state) will be maximally focused for many if not all systems. My attempt at an answer is the triviality measure of the goal, as a counterweight. But it’s still possible in theory to have two goals of equivalent triviality and equivalent focus; in that case I don’t really know yet how to “choose”.