I think the key to approaches like this is to eschew pre-existing, complex concepts like “human flourishing” and look for a definition of Good Things that is actually amenable to constructing an agent that Does Good Things. There’s no guarantee that this would lead anywhere; it relies on some weak form of moral realism. But an AGI that follows some morality-you-largely-agree-with by its very structure is a lot more appealing to me than an AGI that dutifully maximizes the morality-you-punched-into-its-utility-function-at-bootup, appealing enough that I think it’s worth wading into moral philosophy to see if the idea pans out.
I think the key to approaches like this is to eschew pre-existing, complex concepts like “human flourishing” and look for a definition of Good Things that is actually amenable to constructing an agent that Does Good Things. There’s no guarantee that this would lead anywhere; it relies on some weak form of moral realism. But an AGI that follows some morality-you-largely-agree-with by its very structure is a lot more appealing to me than an AGI that dutifully maximizes the morality-you-punched-into-its-utility-function-at-bootup, appealing enough that I think it’s worth wading into moral philosophy to see if the idea pans out.