I think the crux of this post has been correctly argued against by other people in this comment section, but I want to address a small part of your post:
Presumably, the goals explain the extent to which some humans exhibit instrumental convergence, and the rest of the utility function explains why we haven’t yet tiled the universe with money.
No, the reason we haven’t yet tiled the universe with money is that we don’t desire money for its own sake, only becaue it is an instrumental value that helps us achieve our goals. If we could tile the universe, we would tile the universe with happy humans. The problem is that we can’t tile the universe.
If we could tile the universe, we would tile the universe with happy humans.
What, you would? I sure as hell wouldn’t. The “money” example wasn’t meant to be entirely serious, although I suspect that many people see money as an intrinsic reward—they collect a bunch and then don’t really know what to do with it.
I think the crux of this post has been correctly argued against by other people in this comment section, but I want to address a small part of your post:
No, the reason we haven’t yet tiled the universe with money is that we don’t desire money for its own sake, only becaue it is an instrumental value that helps us achieve our goals. If we could tile the universe, we would tile the universe with happy humans. The problem is that we can’t tile the universe.
What, you would? I sure as hell wouldn’t. The “money” example wasn’t meant to be entirely serious, although I suspect that many people see money as an intrinsic reward—they collect a bunch and then don’t really know what to do with it.
To clarify, I am firmly anti-wireheading.