“Von Neumann-Morgenstern axioms” Isn’t it interesting that...Morgenstern was influenced by the second and third generation ‘Austrian’ economists (including Mises directly) and then he gets associated with AI due to his work with Von Neumann; which is frequently correlated itself with laissez-faire, libertarian and techno-commercialism?
To actually address the post, I do agree that utility functions are ordinal but not cardinal. I’m not sure even AI could have a cardinal utility function, since to be a meaningful ‘value’ it most motivate action; its theoretical mathematical relationship to other utility-gains and losses is irrelevant to this actual action. Likewise, a ‘mathematical’ or cardinal utility function can just as easily be described as a psychological or functional ‘system’ utilized, which itself would seem irrelevant to the ranking of values actually involved in purposeful action.
Sorry for the delay, I just checked this: I think actual morality tends to systematically bias behaviour and ideas about ‘social’ life which are contrary to fact and create all sorts of personal and interpersonal problems. I also think it gives far too strong a ‘presumption’ towards the benevolence of do-gooders, the sanity of ‘sticking to your guns, come what may’ and the wisdom of the popular.
There is a more general problem with cognitive dissonance and idea-consistency, due to the literal nonsensicality of most moral claims and sentiments. I also see that the alleged ‘gains’ from morality are frequently self-inflated, if not false to begin with; while the alternative—intellectual consistency and a recognition of purposeful action as aimed at subjective satisfaction—is vastly underrated, even by people of a ‘libertarian’ bent.