What is missing here is an argument that the VNM theorem does have important implications in settings where its assumptions are not true. Nobody has made this argument. I agree it’s suggestive, but that’s very far from demonstrating that AGIs will necessarily be ruthlessly maximising some simple utility function.
“obviously we don’t expect a superintelligent AI to be predictably stupid in the way Eliezer lines out”
Eliezer argued that superintelligences will have certain types of goals, because of the VNM theorem. If they have different types of goals, then behaviour which violates VNM is no longer “predictably stupid”. For example, if I have a deontological goal, then maybe violating VNM is the best strategy.
What is missing here is an argument that the VNM theorem does have important implications in settings where its assumptions are not true. Nobody has made this argument. I agree it’s suggestive, but that’s very far from demonstrating that AGIs will necessarily be ruthlessly maximising some simple utility function.
“obviously we don’t expect a superintelligent AI to be predictably stupid in the way Eliezer lines out”
Eliezer argued that superintelligences will have certain types of goals, because of the VNM theorem. If they have different types of goals, then behaviour which violates VNM is no longer “predictably stupid”. For example, if I have a deontological goal, then maybe violating VNM is the best strategy.