In other words the VNM theorem says that our AGI has to have a utility function
Still nope. The VNM theorem says that if our AGI sticks to VNM axioms then a utility function describing its preferences exists. Exists somewhere in the rather vast space of mathematical functions. The theorem doesn’t say that the AGI “has” it—neither that it knows it, nor that it can calculate it.
Still nope. The VNM theorem says that if our AGI sticks to VNM axioms then a utility function describing its preferences exists. Exists somewhere in the rather vast space of mathematical functions. The theorem doesn’t say that the AGI “has” it—neither that it knows it, nor that it can calculate it.
That’s what I meant.