I fully agree that realizability is needed here. In practice, for the research I’m doing, I’m defining the desired utility as being defined by a constructive process. Therefore the correct human preference set is in there by definition. This requires that the set of possible utilities be massive enough that we’re confident we didn’t miss anything. Then, because the process is constructive, it has to be realizable once we’ve defined the “normative assumptions” that map observations to updates of value functions.
My current intuition is that thinking in terms of non-realizable epistemology will give a more robust construction process, even though the constructive way of thinking justifies a kind of realizability assumption. This is partly because it allows us to do without the massive-enough set of hypotheses (which one may have to do without in practice), but also because it seems closer to the reality of “humans don’t really have a utility function, not exactly”.
However, I think I haven’t sufficiently internalized your point about utility being defined by a constructive process, so my opinion on that may change as I think about it more.
My current intuition is that thinking in terms of non-realizable epistemology will give a more robust construction process, even though the constructive way of thinking justifies a kind of realizability assumption. This is partly because it allows us to do without the massive-enough set of hypotheses (which one may have to do without in practice), but also because it seems closer to the reality of “humans don’t really have a utility function, not exactly”.
However, I think I haven’t sufficiently internalized your point about utility being defined by a constructive process, so my opinion on that may change as I think about it more.