[The following is a musing that might or might not be adding anything.]
As a result, we evolved two internal mechanisms: preference alteration (my own phrase) and preference falsification. Preference alteration is where someone’s preferences actually change according to the social reward gradient, and preference falsification is acting in public according to the reward gradient but not doing so in private. The amounts of preference alteration and preference falsification can vary between individuals. (We have preference alteration because preference falsification is cognitively costly, and we have preference falsification because preference alteration is costly in terms of physical resources.)
One thing that comes to mind here is framing myself as a mesa-optimizer in a (social) training process. Insofar as the training process worked, and I was successfully aligned, my values are the values of the social gradient. Or the I might be an unaligned optimizer intending to execute a treacherous turn (though in this context, the “treacherous turn” is not a discreet moment when I change my actions, but rather a continual back-and-forth between serving selfish interests and serving the social morality, depending on the circumstances).
“Feelings of guilt” is what preference alteration feels like from the inside.
I’m not sure that that is always what it feels like. I can feel pride at my moral execution.
[The following is a musing that might or might not be adding anything.]
One thing that comes to mind here is framing myself as a mesa-optimizer in a (social) training process. Insofar as the training process worked, and I was successfully aligned, my values are the values of the social gradient. Or the I might be an unaligned optimizer intending to execute a treacherous turn (though in this context, the “treacherous turn” is not a discreet moment when I change my actions, but rather a continual back-and-forth between serving selfish interests and serving the social morality, depending on the circumstances).
I’m not sure that that is always what it feels like. I can feel pride at my moral execution.