My claim is mostly that real world intelligent systems do not have values that can be well described by a single fixed utility function over agent states.
I do not see this answer as engaging with that claim at all.
If you define utility functions over agent histories, then everything is an expected utility maximiser for the function that assigns positive utility to whatever action the agent actually took and zero utility to every other action.
I think such a definition of utility function is useless.
If however you define utility functions over agent states, then your hypothesis doesn’t engage with my claim at all. The reason that real world intelligent systems aren’t utility functions isn’t because the utility function is too big to fit inside them or because of incomplete knowledge.
My claim is that no such utility function exists that adequately describes the behaviour of real world intelligent systems.
I am claiming that there is no such mathematical object, no single fixed utility function over agent states that can describe the behaviour of humans or sophisticated animals.
Sorry, I guess I didn’t make the connection to your post clear. I substantially agree with you that utility functions over agent-states aren’t rich enough to model real behavior. (Except, maybe, at a very abstract level, a la predictive processing? (which I don’t understand well enough to make the connection precise)).
Utility functions over world-states—which is what I thought you meant by ‘states’ at first—are in some sense richer, but I still think inadequate.
And I agree that utility functions over agent histories are too flexible.
I was sort of jumping off to a different way to look at value, which might have both some of the desirable coherence of the utility-function-over-states framing, but without its rigidity.
And this way is something like, viewing ‘what you value’ or ‘what is good’ as something abstract, something to be inferred, out of the many partial glimpses of it we have in the form of our extant values.
My claim is mostly that real world intelligent systems do not have values that can be well described by a single fixed utility function over agent states.
I do not see this answer as engaging with that claim at all.
If you define utility functions over agent histories, then everything is an expected utility maximiser for the function that assigns positive utility to whatever action the agent actually took and zero utility to every other action.
I think such a definition of utility function is useless.
If however you define utility functions over agent states, then your hypothesis doesn’t engage with my claim at all. The reason that real world intelligent systems aren’t utility functions isn’t because the utility function is too big to fit inside them or because of incomplete knowledge.
My claim is that no such utility function exists that adequately describes the behaviour of real world intelligent systems.
I am claiming that there is no such mathematical object, no single fixed utility function over agent states that can describe the behaviour of humans or sophisticated animals.
Such a function does not exist.
Sorry, I guess I didn’t make the connection to your post clear. I substantially agree with you that utility functions over agent-states aren’t rich enough to model real behavior. (Except, maybe, at a very abstract level, a la predictive processing? (which I don’t understand well enough to make the connection precise)).
Utility functions over world-states—which is what I thought you meant by ‘states’ at first—are in some sense richer, but I still think inadequate.
And I agree that utility functions over agent histories are too flexible.
I was sort of jumping off to a different way to look at value, which might have both some of the desirable coherence of the utility-function-over-states framing, but without its rigidity.
And this way is something like, viewing ‘what you value’ or ‘what is good’ as something abstract, something to be inferred, out of the many partial glimpses of it we have in the form of our extant values.