(shrug) Translating the terms is rather beside my point here.
If the AGI is using these things to choose among possible future worlds, then I expect it to be able to teach me to choose among possible future worlds more like it does than I would without that explanation.
I’m happy to call those things goals, ethics, morality, etc., even if those words don’t capture what the AGI means by them. (I don’t know that they really capture what I mean by them either, come to that.) Perhaps I would do better to call them “groom” or “fleem” or “untranslatable1” or refer to them by means of a specific shade of orange. I don’t know; but as I say, I don’t really care; terminology is largely independent of explanation.
But, sure, if you expect that it’s incapable of doing that, then our expectations differ.
I’ll note that my expectations don’t depend on my having reached a magical threshold, or on everything being explainable to me given enough effort.
(shrug) Translating the terms is rather beside my point here.
If the AGI is using these things to choose among possible future worlds, then I expect it to be able to teach me to choose among possible future worlds more like it does than I would without that explanation.
I’m happy to call those things goals, ethics, morality, etc., even if those words don’t capture what the AGI means by them. (I don’t know that they really capture what I mean by them either, come to that.) Perhaps I would do better to call them “groom” or “fleem” or “untranslatable1” or refer to them by means of a specific shade of orange. I don’t know; but as I say, I don’t really care; terminology is largely independent of explanation.
But, sure, if you expect that it’s incapable of doing that, then our expectations differ.
I’ll note that my expectations don’t depend on my having reached a magical threshold, or on everything being explainable to me given enough effort.