Surely the discussion is not about the issue of whether an AI will be able to be sophisticated in forming abstractions—if it is of interest, then presumably it will be.
But the concern discussed here is how to determine beforehand that those abstractions will be formed in a context characterised here as Friendly AI. The concern is to pre-ordain that context before the AI achieves superintelligence.
Thus the limitations of communicating desirable concepts apply.
Surely the discussion is not about the issue of whether an AI will be able to be sophisticated in forming abstractions—if it is of interest, then presumably it will be.
But the concern discussed here is how to determine beforehand that those abstractions will be formed in a context characterised here as Friendly AI. The concern is to pre-ordain that context before the AI achieves superintelligence.
Thus the limitations of communicating desirable concepts apply.