I disagree with your implication and agree with OP: Inner alignment is not yet solved, therefore we don’t know how to make an AGI that is “trying” to do something in particular, and by extension, we don’t know how to make an AGI with a particular utility function. I was actually going to comment that OP is not only correct but uncontroversial, at least among experts. (That doesn’t mean it’s a pointless post, good pedagogy is always welcome.) So I’m surprised and confused by your comment.
I disagree with your implication and agree with OP: Inner alignment is not yet solved, therefore we don’t know how to make an AGI that is “trying” to do something in particular, and by extension, we don’t know how to make an AGI with a particular utility function. I was actually going to comment that OP is not only correct but uncontroversial, at least among experts. (That doesn’t mean it’s a pointless post, good pedagogy is always welcome.) So I’m surprised and confused by your comment.
That is fair enough, I was just confused and thought OP had not heard about it because there wasn’t a hint of it in the post.