One more consideration about “instrumental intelligence”: we left that somewhat under-defined, more like “if I had that utility function, what would I do?” … but it is not clear that this image of “me in the machine” captures what a current or future machine would do. In other words, people who use instrumental intelligence for an image of AI owe us a more detailed explanation of what that would be, given the machines we are creating—not just given the standard theory of rational choice.
One more consideration about “instrumental intelligence”: we left that somewhat under-defined, more like “if I had that utility function, what would I do?” … but it is not clear that this image of “me in the machine” captures what a current or future machine would do. In other words, people who use instrumental intelligence for an image of AI owe us a more detailed explanation of what that would be, given the machines we are creating—not just given the standard theory of rational choice.