This, however, is not the usual meaning of the term “preference.” In the standard usage, this word refers to one’s favored option in a given set of available alternatives, not to the hypothetical most favorable physically possible state of the world (which, as you correctly note, is unlikely to be readily imaginable). If you insist on using the term with this meaning, fair enough; it’s just that your claims sound confusing when you don’t include an explanation about your non-standard usage.
That said, one problem I see with your concept of preference is that, presumably, the actions of the “obsessive world-rewriting robot” are supposed to modify the world around you to make it consistent with your preferences, not to modify your mind to make your preferences consistent with the world. However, it is not at all clear to me whether a meaningful boundary between these two sorts of actions can be drawn.
That said, one problem I see with your concept of preference is that, presumably, the actions of the “obsessive world-rewriting robot” are supposed to modify the world around you to make it consistent with your preferences, not to modify your mind to make your preferences consistent with the world. However, it is not at all clear to me whether a meaningful boundary between these two sorts of actions can be drawn.
Preference in this sense is a rigid designator, defined over the world but not determined by anything in the world, so modifying my mind couldn’t make my preference consistent with the world; a robot implementing my preference would have to understand this.
This, however, is not the usual meaning of the term “preference.” In the standard usage, this word refers to one’s favored option in a given set of available alternatives, not to the hypothetical most favorable physically possible state of the world (which, as you correctly note, is unlikely to be readily imaginable). If you insist on using the term with this meaning, fair enough; it’s just that your claims sound confusing when you don’t include an explanation about your non-standard usage.
That said, one problem I see with your concept of preference is that, presumably, the actions of the “obsessive world-rewriting robot” are supposed to modify the world around you to make it consistent with your preferences, not to modify your mind to make your preferences consistent with the world. However, it is not at all clear to me whether a meaningful boundary between these two sorts of actions can be drawn.
Preference in this sense is a rigid designator, defined over the world but not determined by anything in the world, so modifying my mind couldn’t make my preference consistent with the world; a robot implementing my preference would have to understand this.