As I say, open question. We have only one example of a generally intelligent system, and that’s not even very intelligent. We have no clue how to extend or compare that to other types.
It does seem like VNM-rational agents will be better than non-rational agents at achieving their goals. It’s unclear if that’s a nudge to make agents move toward VNM-rationality as they get more capable, or a filter to advantage VNM-rational agents in competition to power. Or a non-causal observation, because goals are orthogonal to power.
Yeah, I think the preconditions of VNM straightforwardly just don’t apply to generally intelligent systems.
As I say, open question. We have only one example of a generally intelligent system, and that’s not even very intelligent. We have no clue how to extend or compare that to other types.
It does seem like VNM-rational agents will be better than non-rational agents at achieving their goals. It’s unclear if that’s a nudge to make agents move toward VNM-rationality as they get more capable, or a filter to advantage VNM-rational agents in competition to power. Or a non-causal observation, because goals are orthogonal to power.