I think that there is hidden assumption that utility function is simple, so it could be easily calculated for any given position. So we have interaction of two algorithms: one extremely simple, utility function, and another is extremely complex (AGI). Most problems like paperclip maximiser results of such interaction.
The question which arise here, is it possible that utility function also will be very complex? For example as complex, as narrow AI? Could it help us in creating Friendly AI? Is known complexity of human values the same thing?
I think that there is hidden assumption that utility function is simple, so it could be easily calculated for any given position. So we have interaction of two algorithms: one extremely simple, utility function, and another is extremely complex (AGI). Most problems like paperclip maximiser results of such interaction.
The question which arise here, is it possible that utility function also will be very complex? For example as complex, as narrow AI? Could it help us in creating Friendly AI? Is known complexity of human values the same thing?