A common theme- the transhuman AI is not yet so transhuman that it can identify actions make people happy and is moral.
If the AI knows only what we can define, and is only correct about what we can describe correctly, then it is not yet transhuman. If an AI does become transhuman, it will presumably know what we want better than we do.
There is a difference between an excessively effective optimizer and intelligence.
A common theme- the transhuman AI is not yet so transhuman that it can identify actions make people happy and is moral.
If the AI knows only what we can define, and is only correct about what we can describe correctly, then it is not yet transhuman. If an AI does become transhuman, it will presumably know what we want better than we do.
There is a difference between an excessively effective optimizer and intelligence.