Nitpicky edit request: your comment contains some typos that make it a bit hard to parse (“be other”, “we it”). (So apologies if my reaction misunderstands your point.)
[Assuming that the opposite of the natural abstraction hypothesis is true—ie, not just that “not all powerful AIs share ontology with us”, but actually “most powerful AIs don’t share ontology with us”:] I also expect that an AI with superior ontology would be able to answer your questions about its ontology, in a way that would make you feel like[1] you understand what is happening. But that isn’t the same as being able to control the AI’s actions, or being able to affect its goal specification in a predictable way (to you). You totally wouldn’t be able to do that.
([Vague intuition, needs work] I suspect that if you had a method for predictably-to-you translating from your ontology to the AI’s ontology, then this could be used to prove that you can easily find a powerful AI that shares an ontology with us. Because that AI could be basically thought of as using our ontology.)
Though note that unless you switched to some better ontology, you wouldn’t actually understand what is going on, because your ontology is so bogus that it doesn’t even make sense to talk about “you understanding [stuff]”. This might not be true for all kinds of [stuff], though. EG, perhaps our understanding of set theory is fine while our understanding of agency, goals, physics, and whatever else, isn’t.
Nitpicky edit request: your comment contains some typos that make it a bit hard to parse (“be other”, “we it”). (So apologies if my reaction misunderstands your point.)
[Assuming that the opposite of the natural abstraction hypothesis is true—ie, not just that “not all powerful AIs share ontology with us”, but actually “most powerful AIs don’t share ontology with us”:]
I also expect that an AI with superior ontology would be able to answer your questions about its ontology, in a way that would make you feel like[1] you understand what is happening. But that isn’t the same as being able to control the AI’s actions, or being able to affect its goal specification in a predictable way (to you). You totally wouldn’t be able to do that.
([Vague intuition, needs work] I suspect that if you had a method for predictably-to-you translating from your ontology to the AI’s ontology, then this could be used to prove that you can easily find a powerful AI that shares an ontology with us. Because that AI could be basically thought of as using our ontology.)
Though note that unless you switched to some better ontology, you wouldn’t actually understand what is going on, because your ontology is so bogus that it doesn’t even make sense to talk about “you understanding [stuff]”. This might not be true for all kinds of [stuff], though. EG, perhaps our understanding of set theory is fine while our understanding of agency, goals, physics, and whatever else, isn’t.