Dumb solution: an FAI could have a sense of justice which downweights the utility function of people who are killing and/or procreating to game their representation in AI’s utility function, or something like that do disincentivize it. (It’s dumb because I don’t know how to operationalize justice; maybe enough people would not cheat and want to punish the cheaters that the FAI would figure that out.)
Also, given what we mostly believe about moral progress, I think defining morality in terms of the CEV of all people who ever lived is probably okay… they’d probably learn to dislike slavery in the AI’s simulation of them.
Dumb solution: an FAI could have a sense of justice which downweights the utility function of people who are killing and/or procreating to game their representation in AI’s utility function, or something like that do disincentivize it. (It’s dumb because I don’t know how to operationalize justice; maybe enough people would not cheat and want to punish the cheaters that the FAI would figure that out.)
Also, given what we mostly believe about moral progress, I think defining morality in terms of the CEV of all people who ever lived is probably okay… they’d probably learn to dislike slavery in the AI’s simulation of them.