The problem then becomes: you do you know it is intelligent—if giving it intelligence tests no longer works.
You put it in a variety of environments and see if they tend to look similar after a while. It’s easier if you have a goal to test against, but as long as it’s optimizing some utility function in a variety of environments, it’s intelligent.
The problem arises when it isn’t doing that. Say you tell the superintelligence to sit still and do nothing. It’s a meditating superintelligence—but you can’t easily determine that until after it has stopped meditating.
You put it in a variety of environments and see if they tend to look similar after a while. It’s easier if you have a goal to test against, but as long as it’s optimizing some utility function in a variety of environments, it’s intelligent.
The problem arises when it isn’t doing that. Say you tell the superintelligence to sit still and do nothing. It’s a meditating superintelligence—but you can’t easily determine that until after it has stopped meditating.