One way that comes to mind is to use the constructive VNM utility theorem proof. The construction is going to be approximate because the system’s rationality is. So next things to study include in what way the rationality is approximate, and how well this and other constructions preserve this (and other?) approximations.
Oh, and isn’t inverse reinforcement learning about this?
See my reply to ricraz’s comment for my thoughts on using VNM utility theorem in general. The use you suggest could work, but if we lean on VNM then the hard part of the problem is backing out the agent’s internal probabilistic model.
IRL is about this, but the key difference is that it black-boxes the agent. It doesn’t know what the agent’s internal governing equations look like, it just sees the outputs.
One way that comes to mind is to use the constructive VNM utility theorem proof. The construction is going to be approximate because the system’s rationality is. So next things to study include in what way the rationality is approximate, and how well this and other constructions preserve this (and other?) approximations.
Oh, and isn’t inverse reinforcement learning about this?
See my reply to ricraz’s comment for my thoughts on using VNM utility theorem in general. The use you suggest could work, but if we lean on VNM then the hard part of the problem is backing out the agent’s internal probabilistic model.
IRL is about this, but the key difference is that it black-boxes the agent. It doesn’t know what the agent’s internal governing equations look like, it just sees the outputs.