Here are two versions of “an AGI will understand very well what I mean”:
Given things in my world model / ontology, the AGI will know which things they translate to in its own world model / ontology, such that the referents (the things “in the real world” being pointed at from our respective models) are essentially coextensive.
For any behaviour I could exhibit (such as pressing a button, or expressing contentment with having reached common understanding in a dialogue) that, for me, turns on the words being used, the AGI is very good at predicting my behaviours conditional on the words I’m using, or causing me to exhibit behaviours by using words itself.
Is version 1 something you get from more and more competence and generality on version 2? I think version 1 is more like the ideal version of “the AGI understands what I mean”, but is more confused (because I’m having to rely on concepts like “know” and “referent” and “translate”).
I think Richard has stated that we can expect an AGI to understand what I mean, in version 2 sense, and either equivocates between the versions or presumes version 2 implies version 1. I think Adam is claiming that version 2 might not imply version 1, or pointing out that there’s still an argument missing there or problem to be solved there.
Here are two versions of “an AGI will understand very well what I mean”:
Given things in my world model / ontology, the AGI will know which things they translate to in its own world model / ontology, such that the referents (the things “in the real world” being pointed at from our respective models) are essentially coextensive.
For any behaviour I could exhibit (such as pressing a button, or expressing contentment with having reached common understanding in a dialogue) that, for me, turns on the words being used, the AGI is very good at predicting my behaviours conditional on the words I’m using, or causing me to exhibit behaviours by using words itself.
Is version 1 something you get from more and more competence and generality on version 2? I think version 1 is more like the ideal version of “the AGI understands what I mean”, but is more confused (because I’m having to rely on concepts like “know” and “referent” and “translate”).
I think Richard has stated that we can expect an AGI to understand what I mean, in version 2 sense, and either equivocates between the versions or presumes version 2 implies version 1. I think Adam is claiming that version 2 might not imply version 1, or pointing out that there’s still an argument missing there or problem to be solved there.