Humans are much better at communicating with less intelligent beings than fish or cattle or chimps are.
You are right that we are certainly able to convey a small simple subset of our goals, desires and motivations to some complex enough animals. You would probably also agree that most of what makes us human can never be explained to a dog or a cat, no matter how hard we try. We appear to them like members of their own species who sometimes make completely incomprehensible decisions they have no choice but put up with.
“an incredibly smart AGI is incredibly smart, so it will be able to find effective strategies for communicating its goals to us if it so desires.”
This is quite possible. It might give us its dumbed-down version of its 10 commandments which would look to us like an incredible feat of science and philosophy.
Which one of these tendencies will win out when it comes to human-AGI interaction? Beats me.
Right. An optimistic view is that we can understand the explanations, a pessimistic view is that we would only be able to follow instructions (this is not the most pessimistic view by far).
I’m pretty skeptical of naive extrapolation in this domain anyway, given Eliezer’s point that major advances in optimization power are meta-level qualitative shifts, and so we shouldn’t expect trends to be maintained across those shifts.
Indeed, we shouldn’t. I probably phrased my point poorly. What I tried to convey is that because “major advances in optimization power are meta-level qualitative shifts”, confidently proclaiming that an advanced AGI will be able to convey what it thinks to humans is based on the just-world fallacy, not on any solid scientific footing.
You are right that we are certainly able to convey a small simple subset of our goals, desires and motivations to some complex enough animals. You would probably also agree that most of what makes us human can never be explained to a dog or a cat, no matter how hard we try. We appear to them like members of their own species who sometimes make completely incomprehensible decisions they have no choice but put up with.
This is quite possible. It might give us its dumbed-down version of its 10 commandments which would look to us like an incredible feat of science and philosophy.
Right. An optimistic view is that we can understand the explanations, a pessimistic view is that we would only be able to follow instructions (this is not the most pessimistic view by far).
Indeed, we shouldn’t. I probably phrased my point poorly. What I tried to convey is that because “major advances in optimization power are meta-level qualitative shifts”, confidently proclaiming that an advanced AGI will be able to convey what it thinks to humans is based on the just-world fallacy, not on any solid scientific footing.