So an AGI knowing only Q could never learn to express any sentence in P, since if it could, any speaker of Q (potentially any non-improved human) could in principle learn to express sentences in P (given an arbitrarily large amount of resources like time, questions and answers, etc.).
Honestly, I expected you to do a bit more steelmanning with the examples I gave. Or maybe you have, just didn’t post them here, Anyway, does the quote mean that any English sentence can be expressed in Chimp, since we evolved from a common ancestor? If you don’t claim that (I hope you don’t), then where did your logic stop applying to humans and chimps vs AGI and humans? Presumably it’s relying on the Premise 3 that gets us a wrong conclusion in the English/Chimp example, since it is required to construct an unbroken chain of languages. What happened to humans over their evolution which made them create Q out of P where Q is not reducible to P? And if this is possible in the mindless evolutionary process, then would it not be even more likely during intelligence explosion?
If that’s right, than something’s being much more intelligent would (as I think someone else mentioned) just lead to very hard to understand instrumental goals. It would have no direct relationship with terminal goals.
I don’t understand this point. I would expect the terminal goals evolve as the evolving intelligence understands more and more about the world. For example, for many people here the original terminal goal was, ostensibly, “serve God”. Then they stopped believing and now their terminal goal is more like “do good”. Similarly, I would expect an evolving AGI to adjust its terminal goals as the ones it had before are obsoleted, not because they have been reached, but because they become meaningless.
Anyway, does the quote mean that any English sentence can be expressed in Chimp, since we evolved from a common ancestor?
No, I said nothing about evolving from a common ancestor. The process of biological variation, selection, and retention of genes seems to be to be entirely irrelevant to this issue, since we don’t know languages in virtue of having specific sets of genes. We know languages by learning them from language-users. You might be referring to homo ancestors that developed language at some time in the past, and the history of linguistic development that led to modern languages. I think my argument does show (if it’s sound) that anything in our linguistic history that qualifies as a language is inter-translatable with a modern language (given arbitrary resources of time interrogation, metaphor, neologism, etc.).
It’s hard to say what qualifies as a language, but then it’s also hard to say when a child goes from being a non-language user to being a language user. It’s certainly after they learn their first word, but it’s not easy to say exactly when. But remember I’m arguing that we can always inter-translate two languages, not that we can some how make the thoughts of a language user intelligible to a non-language user (without making them a language user). This is, incidentally, where I think your AGI:us::us:chimps analogy breaks down. I still see no reason to think it plausible. At any rate, I don’t need to draw a line between those homo that spoke languages and those that did not. I grant that the former could not be understood by the latter. I just don’t think the same goes for languages and ‘meta-languages’.
I would expect the terminal goals evolve as the evolving intelligence understands more and more about the world.
Me too, but that would have nothing to do with intelligence on EY’s definition. If intelligence is optimizing power, then it can’t be used to reevaluate terminal goals. What would it optimize for? It can only be used to reevaluate instrumental goals so as to optimize for satisfying terminal goals. I don’t know how the hell we do reevaluate terminal goals anyway, but we do, so there you go.
For example, for many people here the original terminal goal was, ostensibly, “serve God”. Then they stopped believing and now their terminal goal is more like “do good”.
You might think they just mistook an instrumental goal (‘serve God’) for a terminal goal, when actually they wanted to ‘do good’ all along.
At any rate, I don’t need to draw a line between those homo that spoke languages and those that did not. I grant that the former could not be understood by the latter. I just don’t think the same goes for languages and ‘meta-languages’.
Ah. To me language is just a meta-grunt. That’s why I don’t think it’s different from the next level up. But I guess I don’t have any better arguments than those I have already made and they are clearly not convincing. So I will stop here.
You might think they just mistook an instrumental goal (‘serve God’) for a terminal goal, when actually they wanted to ‘do good’ all along.
Right, you might. Except they may not even had the vocabulary to explain that underlying terminal goal. In this example my interpretation would be that their terminal goal evolved rather than was clarified. Again, I don’t have any better argument, so I will leave it at that.
Honestly, I expected you to do a bit more steelmanning with the examples I gave. Or maybe you have, just didn’t post them here, Anyway, does the quote mean that any English sentence can be expressed in Chimp, since we evolved from a common ancestor? If you don’t claim that (I hope you don’t), then where did your logic stop applying to humans and chimps vs AGI and humans? Presumably it’s relying on the Premise 3 that gets us a wrong conclusion in the English/Chimp example, since it is required to construct an unbroken chain of languages. What happened to humans over their evolution which made them create Q out of P where Q is not reducible to P? And if this is possible in the mindless evolutionary process, then would it not be even more likely during intelligence explosion?
I don’t understand this point. I would expect the terminal goals evolve as the evolving intelligence understands more and more about the world. For example, for many people here the original terminal goal was, ostensibly, “serve God”. Then they stopped believing and now their terminal goal is more like “do good”. Similarly, I would expect an evolving AGI to adjust its terminal goals as the ones it had before are obsoleted, not because they have been reached, but because they become meaningless.
No, I said nothing about evolving from a common ancestor. The process of biological variation, selection, and retention of genes seems to be to be entirely irrelevant to this issue, since we don’t know languages in virtue of having specific sets of genes. We know languages by learning them from language-users. You might be referring to homo ancestors that developed language at some time in the past, and the history of linguistic development that led to modern languages. I think my argument does show (if it’s sound) that anything in our linguistic history that qualifies as a language is inter-translatable with a modern language (given arbitrary resources of time interrogation, metaphor, neologism, etc.).
It’s hard to say what qualifies as a language, but then it’s also hard to say when a child goes from being a non-language user to being a language user. It’s certainly after they learn their first word, but it’s not easy to say exactly when. But remember I’m arguing that we can always inter-translate two languages, not that we can some how make the thoughts of a language user intelligible to a non-language user (without making them a language user). This is, incidentally, where I think your AGI:us::us:chimps analogy breaks down. I still see no reason to think it plausible. At any rate, I don’t need to draw a line between those homo that spoke languages and those that did not. I grant that the former could not be understood by the latter. I just don’t think the same goes for languages and ‘meta-languages’.
Me too, but that would have nothing to do with intelligence on EY’s definition. If intelligence is optimizing power, then it can’t be used to reevaluate terminal goals. What would it optimize for? It can only be used to reevaluate instrumental goals so as to optimize for satisfying terminal goals. I don’t know how the hell we do reevaluate terminal goals anyway, but we do, so there you go.
You might think they just mistook an instrumental goal (‘serve God’) for a terminal goal, when actually they wanted to ‘do good’ all along.
Ah. To me language is just a meta-grunt. That’s why I don’t think it’s different from the next level up. But I guess I don’t have any better arguments than those I have already made and they are clearly not convincing. So I will stop here.
Right, you might. Except they may not even had the vocabulary to explain that underlying terminal goal. In this example my interpretation would be that their terminal goal evolved rather than was clarified. Again, I don’t have any better argument, so I will leave it at that.
I see. If that is true, then I can’t dispute your point (for more than one reason).