Utility functions are very useful for solving decision problems with simple objectives. Human preference is not one of these, but we can often fit a utility function that approximately captures it in a particular situation, which is useful for computing informed suggestions for decisions. The model of one’s preference that informs fitting of utility functions to it for use in contexts of particular decision problems could also be called a model of one’s utility function, but that terminology would be misleading.
The error is forgetting that on human level, all utility functions you can work with are hopelessly simplified approximations, maps of some notion of actual preference, and even an understanding of all these maps considered altogether is a hopelessly simplified approximation, not preference itself. It’s not even useful to postulate that preference is a utility function, as this is not the form that is visible in practice when drawing its maps. Still, having maps for a thing clarifies what it is, better than not having any maps at all, and better yet when maps stop getting confused for the thing itself.
I thought I agreed but upon rereading your comment I am no longer sure. As you say, the notion of a utility function implies a consistent mapping between world states and utility valuations, which is something that humans do not do in practice, and cannot do even in principle because of computational limits.
But I am not sure I follow the very last bit. Surely the best map of the dath ilan parable is just a matrix, or table, describing all the possible outcomes, with degrees of distinction provided to whatever level of detail the subject considers relevant. This, I think, is the most practical and useful amount of compression. Compress further, into a “utility function”, and you now have the equivalent of a street map that includes only topology but without street names, if you’ll forgive the metaphor.
Further, if we aren’t at any point multiplying utilities by probabilities in this thought experiment, one has to ask why you would even want utilities in the first place, rather than simply ranking the outcomes in preference order and picking the best one.
Utility functions are very useful for solving decision problems with simple objectives. Human preference is not one of these, but we can often fit a utility function that approximately captures it in a particular situation, which is useful for computing informed suggestions for decisions. The model of one’s preference that informs fitting of utility functions to it for use in contexts of particular decision problems could also be called a model of one’s utility function, but that terminology would be misleading.
The error is forgetting that on human level, all utility functions you can work with are hopelessly simplified approximations, maps of some notion of actual preference, and even an understanding of all these maps considered altogether is a hopelessly simplified approximation, not preference itself. It’s not even useful to postulate that preference is a utility function, as this is not the form that is visible in practice when drawing its maps. Still, having maps for a thing clarifies what it is, better than not having any maps at all, and better yet when maps stop getting confused for the thing itself.
I thought I agreed but upon rereading your comment I am no longer sure. As you say, the notion of a utility function implies a consistent mapping between world states and utility valuations, which is something that humans do not do in practice, and cannot do even in principle because of computational limits.
But I am not sure I follow the very last bit. Surely the best map of the dath ilan parable is just a matrix, or table, describing all the possible outcomes, with degrees of distinction provided to whatever level of detail the subject considers relevant. This, I think, is the most practical and useful amount of compression. Compress further, into a “utility function”, and you now have the equivalent of a street map that includes only topology but without street names, if you’ll forgive the metaphor.
Further, if we aren’t at any point multiplying utilities by probabilities in this thought experiment, one has to ask why you would even want utilities in the first place, rather than simply ranking the outcomes in preference order and picking the best one.