I don’t know of any formal arguments that predict that all or most future AI systems are purely expected utility maximizers. I suspect most don’t believe that to be the case in any simple way.
I do know of a very powerful argument (a proof, in fact) that if an agent’s goal structure is complete, transitively consistent, continuous, and independent of irrelevant alternatives, then it will be consistent with an expected-utility-maximizing model. See https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem
The open question remains, since humans do not meet these criteria, whether more powerful forms of intelligence are more likely to do so.
As I say, open question. We have only one example of a generally intelligent system, and that’s not even very intelligent. We have no clue how to extend or compare that to other types.
It does seem like VNM-rational agents will be better than non-rational agents at achieving their goals. It’s unclear if that’s a nudge to make agents move toward VNM-rationality as they get more capable, or a filter to advantage VNM-rational agents in competition to power. Or a non-causal observation, because goals are orthogonal to power.
I don’t know of any formal arguments that predict that all or most future AI systems are purely expected utility maximizers. I suspect most don’t believe that to be the case in any simple way.
I do know of a very powerful argument (a proof, in fact) that if an agent’s goal structure is complete, transitively consistent, continuous, and independent of irrelevant alternatives, then it will be consistent with an expected-utility-maximizing model. See https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem
The open question remains, since humans do not meet these criteria, whether more powerful forms of intelligence are more likely to do so.
Yeah, I think the preconditions of VNM straightforwardly just don’t apply to generally intelligent systems.
As I say, open question. We have only one example of a generally intelligent system, and that’s not even very intelligent. We have no clue how to extend or compare that to other types.
It does seem like VNM-rational agents will be better than non-rational agents at achieving their goals. It’s unclear if that’s a nudge to make agents move toward VNM-rationality as they get more capable, or a filter to advantage VNM-rational agents in competition to power. Or a non-causal observation, because goals are orthogonal to power.