AI agents are designed using an agency abstraction. The notion of an AI “having a utility function” itself only has meaning relative to an agency abstraction. There is no such thing as a “real agent” independent of some concept of agency.
All the agency abstractions I know of permit taking one of some specified set of actions at each time step, which can easily be defined to include the “twitch” action. If you disagree with my claim, you can try formalizing a natural one that doesn’t have this property. (There are trivial ways to restrict the set of actions, but then you could use a utility function to rationalize “twitch if you can, take the lexicographically first action you can otherwise”)
Sorry, I wasn’t clear enough. What is the process which both:
Sends the signal to the motor control to twitch, and
Infers that it could break or be interfered with, and sends signals to the motor controls that cause it to be in a universe-state where it is less likely to break or be interfered with?
I claim that for any such reasonable process, if there is a notion of a “goal” in this process, I can create a goal that rationalizes the “always-twitch” policy. If I put in the goal that I construct into the program that you suggest, the policy always twitches, even if it infers that it could break or be interfered with.
The “reasonable” constraint is to avoid processes like “Maximize expected utility, except in the case where you would always twitch, in that case do something else”.
POMDP is an abstraction. Real agents can be interfered with.
AI agents are designed using an agency abstraction. The notion of an AI “having a utility function” itself only has meaning relative to an agency abstraction. There is no such thing as a “real agent” independent of some concept of agency.
All the agency abstractions I know of permit taking one of some specified set of actions at each time step, which can easily be defined to include the “twitch” action. If you disagree with my claim, you can try formalizing a natural one that doesn’t have this property. (There are trivial ways to restrict the set of actions, but then you could use a utility function to rationalize “twitch if you can, take the lexicographically first action you can otherwise”)
How do you imagine the real agent working? Can you describe the process by which it chooses actions?
Presumably twitching requires sending a signal to a motor control and the connection here can be broken
Sorry, I wasn’t clear enough. What is the process which both:
Sends the signal to the motor control to twitch, and
Infers that it could break or be interfered with, and sends signals to the motor controls that cause it to be in a universe-state where it is less likely to break or be interfered with?
I claim that for any such reasonable process, if there is a notion of a “goal” in this process, I can create a goal that rationalizes the “always-twitch” policy. If I put in the goal that I construct into the program that you suggest, the policy always twitches, even if it infers that it could break or be interfered with.
The “reasonable” constraint is to avoid processes like “Maximize expected utility, except in the case where you would always twitch, in that case do something else”.