FWIW I am both pro-utility and anti-utility at the same time: I think your AGI utility hypothesis and the ASI utility hypothesis are basically correct, but think the human utility hypothesis is wrong (humans can’t be adequately modeled by utility functions for the purposes of alignment, even if they can be modeled by them adequately for other purposes), and as a consequence worry that CEV might not be possible depending on what level of identity preservation is desired (in fact I think CEV is largely ill-defined due to identity boundary issues, but that is a separate issue).
FWIW I am both pro-utility and anti-utility at the same time: I think your AGI utility hypothesis and the ASI utility hypothesis are basically correct, but think the human utility hypothesis is wrong (humans can’t be adequately modeled by utility functions for the purposes of alignment, even if they can be modeled by them adequately for other purposes), and as a consequence worry that CEV might not be possible depending on what level of identity preservation is desired (in fact I think CEV is largely ill-defined due to identity boundary issues, but that is a separate issue).