AI cognition doesn’t have to use alien concepts to be uninterpretable. We’ve never fully interpreted human cognition, either, and we know that our introspectively accessible reasoning uses human-understandable concepts.
Just because your thoughts are built using your own concepts, does not mean your concepts can describe how your thoughts are computed.
Or:
The existence of a natural-language description of a thought (like “I want ice cream”) doesn’t mean that your brain computed that thought in a way which can be compactly described by familiar concepts.
Conclusion: Even if an AI doesn’t rely heavily on “alien” or unknown abstractions—even if the AI mostly uses human-like abstractions and features—the AI’s thoughts might still be incomprehensible to us, even if we took a lot of time to understand them.
I don’t think the conclusion follows from the premises. People often learn new concepts after studying stuff, and it seems likely (to me) that when studying human cognition, we’d first be confused because our previous concepts weren’t sufficient to understand it, and then slowly stop being confused as we built & understood concepts related to the subject. If an AI’s thoughts are like human thoughts, given a lot of time to understand them, what you describe doesn’t rule out that the AI’s thoughts would be comprehensible.
The mere existence of concepts we don’t know about in a subject doesn’t mean that we can’t learn those concepts. Most subjects have new concepts.
AI cognition doesn’t have to use alien concepts to be uninterpretable. We’ve never fully interpreted human cognition, either, and we know that our introspectively accessible reasoning uses human-understandable concepts.
Just because your thoughts are built using your own concepts, does not mean your concepts can describe how your thoughts are computed.
Or:
The existence of a natural-language description of a thought (like “I want ice cream”) doesn’t mean that your brain computed that thought in a way which can be compactly described by familiar concepts.
Conclusion: Even if an AI doesn’t rely heavily on “alien” or unknown abstractions—even if the AI mostly uses human-like abstractions and features—the AI’s thoughts might still be incomprehensible to us, even if we took a lot of time to understand them.
I don’t think the conclusion follows from the premises. People often learn new concepts after studying stuff, and it seems likely (to me) that when studying human cognition, we’d first be confused because our previous concepts weren’t sufficient to understand it, and then slowly stop being confused as we built & understood concepts related to the subject. If an AI’s thoughts are like human thoughts, given a lot of time to understand them, what you describe doesn’t rule out that the AI’s thoughts would be comprehensible.
The mere existence of concepts we don’t know about in a subject doesn’t mean that we can’t learn those concepts. Most subjects have new concepts.
I agree that with time, we might be able to understand. (I meant to communicate that via “might still be incomprehensible”)