So, intelligent agents can have a wide variety of goals, and any goal is as good as any other.
The second half of this doesn’t seem right to me, or at least is a little unclear. [Things like instrumental convergence could be a value-agnostic way of sorting goals, and Bostrom’s ‘more or less’ qualifier is actually doing some useful work to rule out pathological goals.]
One more consideration about “instrumental intelligence”: we left that somewhat under-defined, more like “if I had that utility function, what would I do?” … but it is not clear that this image of “me in the machine” captures what a current or future machine would do. In other words, people who use instrumental intelligence for an image of AI owe us a more detailed explanation of what that would be, given the machines we are creating—not just given the standard theory of rational choice.
The second half of this doesn’t seem right to me, or at least is a little unclear. [Things like instrumental convergence could be a value-agnostic way of sorting goals, and Bostrom’s ‘more or less’ qualifier is actually doing some useful work to rule out pathological goals.]
One more consideration about “instrumental intelligence”: we left that somewhat under-defined, more like “if I had that utility function, what would I do?” … but it is not clear that this image of “me in the machine” captures what a current or future machine would do. In other words, people who use instrumental intelligence for an image of AI owe us a more detailed explanation of what that would be, given the machines we are creating—not just given the standard theory of rational choice.