But it’s not like happiness is the Tooth Fairy. It’s an honest ingredient in useful models I have of human beings, at some level of abstraction. If you think future decision-makers might decide happiness “isn’t real,” what I hear is that they’re ditching those models of humans that include happiness, and deciding to use models that never mention it or use it.
And I would consider this a fairly straightforward failure to align with my meta-preferences (my preferences about what should count as my preferences). I don’t want to be modeled in ways that don’t include happiness, and this is a crucial part of talking about my preferences, because there is no such thing as human preferences divorced from any model of the world used to represent them.
I agree that some people are imagining that we’ll end up representing human values in some implicitly-determined, or “objective” model of the world. And such a process might end up not thinking happiness is a good model ingredient. To me, this sounds like a solid argument to not do that.
Alright.
But it’s not like happiness is the Tooth Fairy. It’s an honest ingredient in useful models I have of human beings, at some level of abstraction. If you think future decision-makers might decide happiness “isn’t real,” what I hear is that they’re ditching those models of humans that include happiness, and deciding to use models that never mention it or use it.
And I would consider this a fairly straightforward failure to align with my meta-preferences (my preferences about what should count as my preferences). I don’t want to be modeled in ways that don’t include happiness, and this is a crucial part of talking about my preferences, because there is no such thing as human preferences divorced from any model of the world used to represent them.
I agree that some people are imagining that we’ll end up representing human values in some implicitly-determined, or “objective” model of the world. And such a process might end up not thinking happiness is a good model ingredient. To me, this sounds like a solid argument to not do that.