The empowerment maximizer is still trying to erase all practical differences between me and a paperclip maximizer, which is a goal I don’t like.
This threw me off initially because of the use of ‘paperclip maximizer’ as a specific value system. But I do partially agree with the steelmanned version of this which is “erase all practical differences between you and the maximally longtermist version of you”.
Some component of our values/utility is short term non-empowerment hedonic which conflicts with long term optionality and an empowerment AI would only be aligned with the long term component; thus absent identity preservation mechanisms this AI would want us to constantly sacrifice for the long term.
But once again many things that appear hedonic—such as fun—are actually components of empowerment related intrinsic motivation, so if the empowerment AGI was going to change us (say after uploading), it would keep fun or give us some improved version of it.
But I actually already agreed with this earlier:
So in essence you are arguing that you may have discount rate high enough to cause significant conflict between long term and short term utility, and empowerment always favors long term. I largely agree with this, but we can combine long term empowerment with learned human values to cover any short term divergences.
Also it’s worth noting that everything here assumes superhuman AGI. When that is realized it changes everything in the sense that the better versions of ourselves—if we had far more knowledge, time to think, etc—probably would be much more long termist.
This threw me off initially because of the use of ‘paperclip maximizer’ as a specific value system. But I do partially agree with the steelmanned version of this which is “erase all practical differences between you and the maximally longtermist version of you”.
Some component of our values/utility is short term non-empowerment hedonic which conflicts with long term optionality and an empowerment AI would only be aligned with the long term component; thus absent identity preservation mechanisms this AI would want us to constantly sacrifice for the long term.
But once again many things that appear hedonic—such as fun—are actually components of empowerment related intrinsic motivation, so if the empowerment AGI was going to change us (say after uploading), it would keep fun or give us some improved version of it.
But I actually already agreed with this earlier:
Also it’s worth noting that everything here assumes superhuman AGI. When that is realized it changes everything in the sense that the better versions of ourselves—if we had far more knowledge, time to think, etc—probably would be much more long termist.