On further reflection, the wish as expressed by Nick Tarleton above sounds dangerous, because all human morality may either be inconsistent in some sense, or ‘naive’ (failing to account for important aspects of reality we aren’t aware of yet). Human morality changes as our technology and understanding changes, sometimes significantly. There is no reason to believe this trend will stop. I am afraid (genuine fear, not figure of speech) that the quest to properly formalize and generalize human morality for use by a ‘friendly AI’ is akin to properly formalizing and generalizing Ptolemean astronomy.
On further reflection, the wish as expressed by Nick Tarleton above sounds dangerous, because all human morality may either be inconsistent in some sense, or ‘naive’ (failing to account for important aspects of reality we aren’t aware of yet). Human morality changes as our technology and understanding changes, sometimes significantly. There is no reason to believe this trend will stop. I am afraid (genuine fear, not figure of speech) that the quest to properly formalize and generalize human morality for use by a ‘friendly AI’ is akin to properly formalizing and generalizing Ptolemean astronomy.