Isn’t acting maximally intelligently and correctly itself a motivation? The question you are really asking seems to be why an AI is supposed to act maximally intelligently and correctly to achieve world states that are not explicitly or implicitly defined to maximize expected utility. Yet the motivation to act maximally intelligently and correctly will always be given, otherwise you’re not talking about an rational agent.
The problem then becomes: you do you know it is intelligent—if giving it intelligence tests no longer works.
You put it in a variety of environments and see if they tend to look similar after a while. It’s easier if you have a goal to test against, but as long as it’s optimizing some utility function in a variety of environments, it’s intelligent.
The problem arises when it isn’t doing that. Say you tell the superintelligence to sit still and do nothing. It’s a meditating superintelligence—but you can’t easily determine that until after it has stopped meditating.
Isn’t acting maximally intelligently and correctly itself a motivation? The question you are really asking seems to be why an AI is supposed to act maximally intelligently and correctly to achieve world states that are not explicitly or implicitly defined to maximize expected utility. Yet the motivation to act maximally intelligently and correctly will always be given, otherwise you’re not talking about an rational agent.
To act maximally intelligently and correctly could quite easily be an instruction to convert the observable universe into a computer.
You could have a very intellligent agent that acts as though it is completely nuts.
The problem then becomes: you do you know it is intelligent—if giving it intelligence tests no longer works.
However, I think this is a bit of a side-issue.
You put it in a variety of environments and see if they tend to look similar after a while. It’s easier if you have a goal to test against, but as long as it’s optimizing some utility function in a variety of environments, it’s intelligent.
The problem arises when it isn’t doing that. Say you tell the superintelligence to sit still and do nothing. It’s a meditating superintelligence—but you can’t easily determine that until after it has stopped meditating.